40 / 67 · 06 Observability-Driven Testing · Metrics and Alerting← prev⊞ allnext →☰ Read as one page
6.6Alert Design Principles
Alert on symptoms, not causes. Alert on "users are seeing errors" not "CPU is high." High CPU that causes no user impact is not an alert-worthy event.
Use multi-window, multi-burn-rate alerts. Catch both sudden outages and gradual degradation.
Every alert must have a runbook. If there is no documented response procedure, the alert is not ready for production.
Page only for things that need immediate human action. Everything else should be a ticket, a dashboard, or a weekly report.