42 / 62 · 22 Test Strategy & Quality Metrics · Measuring Test Effectiveness← prev⊞ allnext →☰ Read as one page
5.8When Metrics Lie: Goodhart's Law
Goodhart's Law
"When a measure becomes a target, it ceases to be a good measure."
How It Applies to QA Metrics
| Metric Target | Gaming Behavior | Actual Outcome |
|---|---|---|
| "Increase code coverage to 90%" | Writing tests with no assertions that execute code but verify nothing | High coverage, poor test quality |
| "Reduce bug count" | Classifying bugs as "by design" or "won't fix" instead of fixing them | Fewer bugs on paper, same bugs in production |
| "Increase automated test count" | Writing trivial tests (assert true == true) | High count, zero value |
| "Reduce flaky test rate to 0%" | Deleting all intermittently failing tests | Zero flaky tests, less coverage |
| "Zero customer-reported defects" | Making it harder for customers to report bugs | Fewer reports, same defects |
Defending Against Metrics Gaming
- Use composite metrics instead of single metrics. A team that games coverage will be caught by mutation score. A team that games bug count will be caught by customer-reported defects.
- Combine quantitative with qualitative. Pair coverage numbers with code review of test quality.
- Track trends, not targets. "Is coverage improving?" is healthier than "Is coverage above 80%?"
- Review the metrics themselves. Quarterly, ask: "Are these metrics still telling us what we need to know?"
- Make metrics informational, not punitive. When metrics are tied to performance reviews, gaming becomes inevitable.