54 / 67 · 06 Observability-Driven Testing · Alert Design: Detecting Real Problems Without Alert Fatigue← prev⊞ allnext →☰ Read as one page
8.5Alert Hygiene Practices
Weekly Alert Review
Every week, review the past week's alerts:
| Question | Action if "Yes" |
|---|---|
| Did any page not require action? | Downgrade to ticket or dashboard |
| Did any page go unacknowledged > 10 min? | Check routing and on-call assignment |
| Were there > 5 pages from the same alert? | Add deduplication or increase thresholds |
| Did any incident go undetected? | Add a new alert for the gap |
| Were any alerts flapping (firing/resolving repeatedly)? | Add hysteresis or increase for duration |
Alert SLOs
Yes, your alerting system itself should have SLOs:
| Metric | Target |
|---|---|
| False positive rate | < 20% of pages |
| Mean time to acknowledge | < 5 minutes |
| Alert-to-action ratio | > 70% |
| Pages per on-call shift | < 5 |
| Undetected incidents | 0 |
Good alert design is a continuous practice, not a one-time configuration. Treat your alerts with the same rigor you treat your test suite.