6.2Testing What Matters
Anti-Pattern: Chase code coverage percentages. "We have 80% coverage" sounds good but says nothing about whether the right things are covered. The login page has 100% coverage; the payment error handling has 0%.
Pattern: Risk-based test strategy — allocate testing effort proportional to business risk, not code volume.
The Risk Matrix
Plot features on two axes:
| Low Failure Likelihood | High Failure Likelihood | |
|---|---|---|
| High Business Impact | Monitor (stable but critical) | Test heavily (critical and volatile) |
| Low Business Impact | Deprioritize (stable and low-stakes) | Fix the instability (volatile, even if low-stakes) |
Inputs to the risk matrix:
- Business impact mapping — What happens if this feature breaks? Revenue loss? User churn? Regulatory violation?
- Failure history — What broke in the last six months? Features that broke before are more likely to break again
- Code complexity and change frequency — Complex code that changes often is the highest-risk combination
Scoring What to Automate
When deciding what to automate next, score each candidate across the dimensions that predict pain — not just "what is easiest to automate":
Business criticality · frequency of use · defect history · production impact · code churn · feature complexity · manual testing cost · customer visibility · compliance/security risk
The output is a ranked list. "I don't automate what is easiest to automate. I automate what is most expensive to get wrong."
Production Behavior as a Quality Signal
The richest source of risk data is what real users actually do — and it forces a wider definition of quality. A 503 error is a failure, but so is a page that loads perfectly and still leaves the user stuck. Quality is the absence of a gap between what the user expects and how the system behaves.
Do not feed raw production logs to an LLM — they exceed the context window, cost too much in tokens, and slicing them by time tears user sessions apart. Build a reduction pipeline instead:
Raw events → sessionization → flow / funnel / anomaly mining → aggregated signals → small set of examples → LLM/human analysis → QA priorities
- Sessionize first — the unit of analysis is one user's journey, not a 5-minute log window. Mine sessions into flow variants; the dominant revenue-critical flows are exactly what regression must cover (this is process mining).
- Find frustration — rage/dead/error clicks, dwell time above a user's baseline, abandonment after a validation error. These are quality failures no functional test catches.
- Find outliers — rare paths with disproportionate abandonment ("0.4% of users hit this flow, but 63% abandon it") deserve targeted regression or exploratory charters.
- LLM is the analyst, not the log processor — hand the model a 20 KB summary, not 20 TB of logs.
Closing the Feedback Loop
The modern quality loop is circular: requirements → code → tests → production → user behavior → back into requirements. QA does not need to own production monitoring — it needs to consume operational intelligence that other teams already own:
- Product — which journeys make money? where do users abandon?
- Support — top complaints? features users don't understand? (A user who can't find the export button filed no bug and is still a quality problem.)
- SRE / DevOps — noisy services, frequent alerts, worrying latency
- Engineering leads — what code scares you? where do hotfixes happen?
Ask for read-only dashboards (RUM, latency, funnels, drop-offs), then institutionalize a 30-minute weekly Quality Review that turns incidents, support tickets, slow endpoints, and adoption data into automation candidates, exploratory charters, and a re-ranked risk matrix. Even a manual "top 10 flows / top 10 abandoned flows" list usually reveals that the regression suite is aimed at features nobody uses.