8.3Designing Alerts for Common Scenarios

Web Application SLO Alerts

# Best practice: multi-burn-rate SLO alerts
groups:
  - name: web-app-slo
    rules:
      - alert: WebAppHighErrorRate_FastBurn
        expr: |
          (error_ratio_5m > 14.4 * 0.001) and (error_ratio_1h > 14.4 * 0.001)
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Error rate burning budget at 14.4x (2-day exhaustion)"

      - alert: WebAppHighErrorRate_SlowBurn
        expr: |
          (error_ratio_30m > 3 * 0.001) and (error_ratio_6h > 3 * 0.001)
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "Error rate burning budget at 3x (10-day exhaustion)"

Certificate Expiration

      - alert: CertificateExpiringSoon
        expr: |
          (cert_expiry_timestamp_seconds - time()) / 86400 < 14
        labels:
          severity: warning   # ticket, not page
        annotations:
          summary: "TLS certificate expires in {{ $value | humanizeDuration }}"

      - alert: CertificateExpiringCritical
        expr: |
          (cert_expiry_timestamp_seconds - time()) / 86400 < 3
        labels:
          severity: critical  # page -- 3 days is urgent

Deployment Failure

      - alert: DeploymentRollbackDetected
        expr: |
          kube_deployment_status_observed_generation
          != kube_deployment_metadata_generation
        for: 10m
        labels:
          severity: warning   # auto-rollback handled it, but investigate
        annotations:
          summary: "Deployment {{ $labels.deployment }} appears to have rolled back"