6.5Multi-Burn-Rate Alerts

The most important advance in alerting over the past decade is multi-burn-rate alerts. Instead of a single threshold ("error rate > 1%"), they detect both fast-burn (sudden outage) and slow-burn (gradual degradation) problems.

How Burn Rate Works

If your SLO allows 0.1% errors over 30 days, the sustainable error rate is 0.1%. A "burn rate" of 1x means you are consuming your error budget at exactly the sustainable rate.

Burn Rate	What It Means	Budget Duration
1x	Sustainable rate	Budget lasts 30 days
3x	Slow burn	Budget exhausted in 10 days
6x	Moderate burn	Budget exhausted in 5 days
14.4x	Fast burn	Budget exhausted in 2 days

Prometheus Alert Rules

# prometheus-alerting-rules.yaml
groups:
  - name: slo-alerts
    rules:
      # Fast burn: consuming error budget at 14.4x the sustainable rate
      # Will exhaust 30-day budget in 2 days if unchecked
      - alert: HighErrorRateFastBurn
        expr: |
          (
            sum(rate(http_requests_total{status=~"5..", service="checkout"}[5m]))
            /
            sum(rate(http_requests_total{service="checkout"}[5m]))
          ) > (14.4 * 0.001)
          and
          (
            sum(rate(http_requests_total{status=~"5..", service="checkout"}[1h]))
            /
            sum(rate(http_requests_total{service="checkout"}[1h]))
          ) > (14.4 * 0.001)
        for: 2m
        labels:
          severity: critical
          team: checkout
        annotations:
          summary: "Checkout error rate burning budget fast (14.4x)"
          runbook: "https://wiki.internal/runbooks/checkout-high-error-rate"
          dashboard: "https://grafana.internal/d/checkout-slo"

      # Slow burn: consuming at 3x the sustainable rate
      # Will exhaust 30-day budget in 10 days if unchecked
      - alert: HighErrorRateSlowBurn
        expr: |
          (
            sum(rate(http_requests_total{status=~"5..", service="checkout"}[30m]))
            /
            sum(rate(http_requests_total{service="checkout"}[30m]))
          ) > (3 * 0.001)
          and
          (
            sum(rate(http_requests_total{status=~"5..", service="checkout"}[6h]))
            /
            sum(rate(http_requests_total{service="checkout"}[6h]))
          ) > (3 * 0.001)
        for: 15m
        labels:
          severity: warning
          team: checkout
        annotations:
          summary: "Checkout error rate burning budget slowly (3x)"
          runbook: "https://wiki.internal/runbooks/checkout-elevated-errors"

The dual-window technique (short window AND long window) reduces false positives. A brief spike that resolves quickly will trigger the short window but not the long window, preventing unnecessary pages.