1.14Interview Depth Check

Question 1

Prompt: You are asked to generate tests for a checkout API at a new company. You have never seen the codebase. What information do you gather before writing your first prompt? What a strong answer should cover:

• All five elements mapped to information sources
• Prioritizing specification and existing test patterns
• Identifying the tech stack, auth mechanism, and data formats
• Asking about team conventions and anti-patterns Example answer:
• Context: I ask for the tech stack (language, framework, database, runtime version), architecture style (monolith vs microservices), and external service dependencies (payment processor, email service)
• Artifact: I request the OpenAPI schema or Swagger file for the checkout endpoint, plus any acceptance criteria or user stories from the project management tool
• Constraints: I ask to see 2-3 existing test files to understand naming conventions, fixture patterns, assertion style, and which testing libraries are used. I also ask about team anti-patterns -- "What are the things you do not want in tests?"
• Coverage Goals: I ask what the team considers high-priority: is it coverage breadth, error handling depth, or performance scenarios? What is the target negative-to-positive test ratio?
• Output Format: I ask how tests are organized (one file per endpoint? per feature?), what grouping structure is used (describe blocks? test classes?), and whether tests should include inline comments

Question 2

Prompt: You send a prompt and get back 10 tests. Seven are happy-path tests, two have vague assertions, and one references a method that does not exist. Diagnose the prompt issues. What a strong answer should cover:

• Happy-path bias indicates missing coverage goals for negative/error cases
• Vague assertions indicate missing output format specifications
• Hallucinated method indicates missing or incomplete artifact
• Specific prompt fixes for each issue Example answer:
• Seven happy-path tests out of 10: my coverage goals did not specify a negative-to-positive ratio or list specific error scenarios. Fix: add "Include at least 2 negative/error cases per acceptance criterion" and explicitly list the error conditions (missing fields, wrong types, auth failures)
• Vague assertions like "expect(result).toBeDefined()": my output format constraints did not specify assertion depth. Fix: add "Every test must assert on specific field values, not just existence or status codes"
• Hallucinated method: my artifact was incomplete -- either I described the API instead of pasting the schema, or I pasted the schema without resolving $ref references. Fix: paste the actual endpoint schema with all referenced components resolved
• After making these three fixes, I regenerate and expect a much better result. If the second iteration still has issues, I refine further and save the final version as a template

Question 3

Prompt: How do you measure whether a test-generation prompt is "good"? What a strong answer should cover:

• Quantitative metrics: first-run pass rate, deletion rate, negative-to-positive ratio
• Qualitative metrics: assertion quality, style match, coverage completeness
• Comparison against the five-element checklist
• Iteration tracking across refinement cycles Example answer:
• First-run pass rate: what percentage of generated tests compile and pass without modification? Target: 70-85%. Below 60% means the context or constraints are wrong
• Deletion rate: what percentage of tests are deleted during review as tautologies, duplicates, or irrelevant? Target: 10-20%. Above 30% means the coverage goals or artifact are insufficient
• Negative-to-positive ratio: target 2:1. If the ratio is 1:4, the coverage goals need explicit negative case requirements
• Style match: do the generated tests use the correct fixtures, naming convention, and assertion patterns? If not, the constraints or style reference need improvement
• Five-element checklist: I score each prompt element 1-3 (weak/adequate/strong) and target a total of 12+/15. Missing elements are the most common root cause of poor output
• I track these metrics across iterations: if iteration 2 improves pass rate from 60% to 80%, the prompt change was effective

Question 4

Prompt: A junior engineer on your team complains that AI-generated tests are useless because "the AI just makes stuff up." How do you respond? What a strong answer should cover:

• Validating the observation while identifying the root cause
• Explaining that hallucination is a prompt quality issue, not an AI limitation
• Demonstrating the difference between an unstructured and structured prompt
• Offering to help the engineer improve their prompts Example answer:
• The observation is valid -- with poor prompts, AI absolutely generates hallucinated tests. The root cause is not that AI is useless but that the prompt lacked the right information
• I would show the comparison: "Write tests for checkout" produces 2 generic, hallucinated tests. A structured prompt with the OpenAPI schema, existing test patterns, and explicit coverage goals produces 25-35 specific, accurate tests
• The key insight: AI hallucination in test generation is directly proportional to missing context. If the AI invents a method name, it is because you did not provide the real method names. If it uses the wrong framework, you did not specify the right one
• I would offer to pair with the engineer on their next test generation task, walking through the five-element framework together and showing the before/after quality difference
• The goal is to transform their perspective from "AI is unreliable" to "AI output quality is proportional to prompt quality, which I can control"