Atlas › 02 AI-Augmented Test Design › Common AI Test Failures☰ Read as one page
Common AI Test Failures
7.1Why AI-Generated Tests FailAI-generated tests fail in predictable ways. Understanding these failure modes lets you spot them instantly during review, rather than…7.2Failure Mode 1: The Tautology TestA tautology test tests the mock, not the code. It asserts that a value you explicitly set up is returned -- which will always be true…7.3Failure Mode 2: The Happy Path OnlyAI generates 10 tests, all for successful cases. No error handling, no edge cases, no auth failures.7.4Failure Mode 3: The Overly Specific AssertionThe test asserts exact error message text, timestamps, or auto-generated IDs that change between runs.7.5Failure Mode 4: The Hallucinated APIAI invents methods, endpoints, or parameters that do not exist in your codebase.7.6Failure Mode 5: The Non-Deterministic TestThe test depends on time, random data, or external state that changes between runs.7.7Failure Mode 6: The Assertion-Free TestThe test runs code but never asserts anything. It only proves the code does not throw an exception.7.8Failure Mode Summary Table7.9The One-Line Review QuestionFor every AI-generated test, ask this single question:7.10Key TakeawayAI test failures are predictable and classifiable. Learn the six failure modes, and you can review 30 AI-generated tests in 15 minutes with…