42 / 67 · 06 Observability-Driven Testing · Metrics and Alerting← prev⊞ allnext →☰ Read as one page
6.8Alert Fatigue: The Enemy of Observability
Alert fatigue occurs when on-call engineers receive so many alerts that they begin ignoring them. This is the single biggest risk to an observability-driven testing program.
Diagnosing Alert Fatigue
| Symptom | Indicates |
|---|---|
| >10 pages per on-call shift | Too many alerts, insufficient filtering |
| >50% of pages are "no action needed" | Too many false positives |
| Average acknowledgment time > 10 minutes | Engineers are ignoring alerts |
| Snooze rate > 30% | Alerts are not actionable |
Curing Alert Fatigue
- Audit every alert. For each alert, ask: "Did this require immediate human action?" If the answer is consistently "no," downgrade from page to ticket.
- Group correlated alerts. If a single incident triggers 15 alerts, create a meta-alert that groups them.
- Set maintenance windows. During deployments, suppress known transient alerts.
- Review monthly. Track alert-to-action ratio. Target >70% of pages resulting in meaningful action.
Metrics and alerting are the real-time nervous system of your production environment. Well-designed alerts catch problems before users notice; poorly designed alerts create noise that hides real problems.