Library › Book 22 › When Metrics Lie -- Goodhart's Law and QA
When Metrics Lie -- Goodhart's Law and QA
21.1🔒Goodhart's Law"When a measure becomes a target, it ceases to be a good measure."
21.2🔒How Goodhart's Law Applies to QA
21.3🔒The Politics of MetricsMetrics exist in a political context. Understanding this context is essential for using metrics effectively:
21.4🔒Defending Against Metrics Gaming1. Use composite metrics instead of single metrics. A team that games coverage will be caught by mutation score. A team that games bug…
21.5🔒Vanity Metrics vs. Actionable MetricsVanity Metrics (look good, mean little):
21.6🔒Setting Targets and Baselines Without GamingHow to set a baseline: 1. Measure for 3 months without changing anything. This is your baseline. 2. Identify the worst metrics -- these are…
21.7🔒Q&ASelf-Assessment Quiz: Chapter 211. State Goodhart's Law. 2. Give an example of how a code coverage target can be gamed. 3. Why should quality metrics not be tied to…
21.8🔒Key Takeaways- Goodhart's Law is the most important principle in quality metrics: targets invite gaming - Metrics are political -- use them as mirrors…
21.9🔒Hands-On ExercisesExercise 21.1 (Beginner): Identify one metric your team tracks that might be subject to Goodhart's Law. Describe how it could be gamed.
21.10🔒Q&AResume phrasing- Identified and remediated 4 Goodhart's Law vulnerabilities in the organization's quality metrics program, replacing single-metric targets…
21.11🔒Q&ACover letter framingI apply Goodhart's Law as a design principle for metrics programs: if a metric can be gamed, assume it will be. I design composite metrics…
21.12🔒Q&AInterview framing"I approach quality metrics with Goodhart's Law as my first principle: when a measure becomes a target, it ceases to be a good measure. I…
21.13🔒Q&AWhat not to say- "We have 100% code coverage so our quality is excellent" -- this is the most common gaming example; coverage without assertion quality…
21.14🔒Q&AQuestion 1Prompt: Your manager mandates "90% code coverage for all teams." How do you push back constructively? What a strong answer should cover…
21.15🔒Q&AQuestion 2Prompt: A QA engineer on your team files 50 bugs per sprint. Another files 10. Is the first engineer better? How do you evaluate? What a…
21.16🔒Q&AQuestion 3Prompt: You suspect a team is gaming their defect escape rate by reclassifying production bugs as "by design" or "known issue." How do you…