8.2Core Alert Design Principles

1. Alert on Symptoms, Not Causes

Alert on what users experience, not what the infrastructure is doing:

Symptom (Good Alert)	Cause (Bad Alert)
"Error rate > 1% for checkout API"	"CPU > 80% on checkout-pod-3"
"p99 latency > 2s for search"	"Memory usage > 90% on search-worker"
"Zero orders processed in 5 minutes"	"Database connection pool at 95%"

High CPU that causes no user impact is not alert-worthy. Database connection pool at 95% might be perfectly normal under load. Focus on what the user sees.

2. Use Multi-Window, Multi-Burn-Rate Alerts

Instead of a single threshold, use a tiered approach that catches both fast-burn (sudden outage) and slow-burn (gradual degradation):

Alert Type	Short Window	Long Window	For Duration	Severity
Fast burn	5 min	1 hour	2 min	Critical (page)
Moderate	15 min	3 hours	5 min	High (page)
Slow burn	30 min	6 hours	15 min	Warning (ticket)

The dual-window requirement reduces false positives: a brief spike triggers the short window but not the long window, so it does not fire.

3. Every Alert Must Have a Runbook

An alert without a runbook is a question without an answer. The runbook should include:

## Runbook: Checkout Error Rate > SLO

### What this alert means
The checkout service error rate has exceeded the SLO burn rate for the
specified window, indicating a potential reliability issue.

### Impact
Users may be unable to complete purchases. Revenue impact is proportional
to the duration and severity.

### Diagnosis steps
1. Check the error rate dashboard: [link]
2. Check recent deployments: `kubectl rollout history deployment/checkout`
3. Check dependency health: [payment-service dashboard link]
4. Check logs: `query: service=checkout level=error | last 15m`

### Common causes and fixes
- **Recent deployment**: Roll back with `kubectl rollout undo deployment/checkout`
- **Payment provider outage**: Enable fallback payment processor
- **Database connection exhaustion**: Scale up connection pool
- **Rate limiting from downstream**: Check rate limit headers

### Escalation
If not resolved within 30 minutes, escalate to:
- #checkout-team Slack channel
- Checkout team lead (page)

4. Page Only for Immediate Human Action

Action Needed	Notification Type	Response Time
Immediate human intervention	Page (PagerDuty)	Minutes
Fix within a day	Ticket (Jira)	Hours
Awareness, no action	Dashboard / weekly report	Days

If the answer to "what should the on-call do?" is "nothing, it will resolve itself," it should not be a page.