Library › Book 2 › The Six Failure Modes of AI-Generated Tests
The Six Failure Modes of AI-Generated Tests
9.1🔒Why AI Tests Fail in Predictable WaysAI-generated tests fail in predictable, classifiable patterns. These failure modes emerge from how LLMs work: they predict statistically…
9.2🔒Failure Mode 1: The Tautology TestA tautology test tests the mock, not the code. It asserts that a value you set up is returned -- always true regardless of code correctness.
9.3🔒Failure Mode 2: Happy Path OnlyAI generates 10 tests, all for successful cases. No errors, no edge cases, no auth failures.
9.4🔒Failure Mode 3: Overly Specific AssertionsTests assert exact error messages, timestamps, or auto-generated IDs that change between runs.
9.5🔒Failure Mode 4: Hallucinated APIAI invents methods, endpoints, or parameters that do not exist in your codebase.
9.6🔒Failure Mode 5: Non-Deterministic TestsTests depend on time, random data, or external state that changes between runs.
9.7🔒Failure Mode 6: Assertion-Free TestsTests run code but never assert anything.
9.8🔒Failure Mode Summary Table
9.9🔒The One-Line Review QuestionFor every AI-generated test:
9.10🔒Q&ASelf-Assessment Quiz1. What is a tautology test? 2. What percentage of AI-generated suites suffer from happy-path bias? 3. How do you detect hallucinated APIs?…
9.11🔒Key Takeaways- AI test failures are predictable and classifiable into six modes - Tautologies and happy-path bias account for over half of all issues…
9.12🔒ExercisesExercise 9.1 (Beginner): Generate 20 tests and classify each into one of the six failure modes (or "no failure"). What is the distribution?
9.13🔒Career Translation- Developed expertise in identifying the six failure modes of AI-generated tests (tautology, happy-path bias, overly specific assertions…
9.14🔒Q&AInterview Depth CheckPrompt: You receive 25 AI-generated tests and need to classify each by failure mode. Walk me through your process and what you look for…