42 / 87 · 02 AI-Augmented Test Design · Common AI Test Failures← prev⊞ allnext →☰ Read as one page
7.5Failure Mode 4: The Hallucinated API
AI invents methods, endpoints, or parameters that do not exist in your codebase.
# AI invents methods that don't exist
result = client.orders.find_by_status("pending") # find_by_status doesn't exist
user = UserService.get_or_create(email="test@test.com") # get_or_create doesn't exist
response = client.patch("/api/v2/orders/1/cancel") # this endpoint doesn't exist
Why AI generates this: The LLM has seen similar APIs in its training data (Django's get_or_create, Rails' find_by_*) and applies those patterns to your codebase.
How to detect it: Always grep your codebase for any function, method, or endpoint the AI references.
# Quick hallucination check
grep -rn "find_by_status" app/ --include="*.py"
grep -rn "get_or_create" app/ --include="*.py"
grep -rn "/cancel" app/routes/ --include="*.py"
If grep returns nothing, the API is hallucinated. Delete the test or replace the call with the actual API.
How to prevent it: Include the actual function signatures or endpoint list in your prompt context.