9.2Part 1: Measuring AI Test-Suite Quality
The review checklist and quality gates from the previous chapters are per-test judgments. At the suite level, three metrics tell you objectively whether AI generation is helping or quietly degrading your safety net:
| Metric | What It Measures | How to Measure | Healthy Signal |
|---|---|---|---|
| Mutation score | Do the tests actually detect bugs? | mutmut/cosmic-ray (Python), Stryker (JS/TS) | > 70% on critical modules; AI tests converging toward hand-written scores |
| Coverage delta | Did each generation session add real coverage? | Compare --cov reports before/after each session |
Positive delta per session; no drops after refactors |
| Flake rate | Are AI tests deterministic? | Repeat runs (pytest --count=3), CI retry statistics |
< 1% of runs; flat or decreasing trend |
Why these three together: coverage tells you what code the tests touch, mutation score tells you whether the assertions would notice a bug there, and flake rate tells you whether the signal can be trusted run-to-run. A suite can look great on any one of them and still be rotten -- 90% coverage made of tautology tests has a dismal mutation score; a high-mutation-score suite that flakes 5% of the time trains the team to ignore red builds.
Track the Deltas per Generation Session
The practical trick is to measure per AI-generation session, not just globally:
# Before the session: snapshot the baseline
pytest --cov=app --cov-report=json:cov_before.json
# ... generate, curate, integrate tests (see the curation workflow) ...
# After the session: compare
pytest --cov=app --cov-report=json:cov_after.json
python scripts/coverage_delta.py cov_before.json cov_after.json
# Output: "+4.2 percentage points, 312 newly covered lines, 0 lines lost"
Do the same with mutation testing scoped to the new tests only (mutation testing the whole suite on every PR is too slow -- run it nightly, scoped to modules the AI touched). If a generation session adds 30 tests, +0.5% coverage, and no mutation-score improvement, the session produced bulk, not protection -- fix the prompt template, not just the tests.