8.4Alert Testing

Alerts are code. They deserve testing like any other code.

Unit Testing Alert Rules

# test_alert_rules.py
"""
Test Prometheus alerting rules using promtool or programmatic evaluation.
"""
import subprocess
import yaml

def test_fast_burn_alert_fires_on_high_error_rate():
    """Verify the fast-burn alert fires when error rate exceeds 14.4x threshold."""
    test_case = {
        "interval": "1m",
        "input_series": [
            {
                "series": 'http_requests_total{service="checkout",status="500"}',
                "values": "0+10x60"  # 10 errors per minute for 60 minutes
            },
            {
                "series": 'http_requests_total{service="checkout",status="200"}',
                "values": "0+100x60"  # 100 successes per minute
            }
        ],
        "alert_rule_test": [
            {
                "eval_time": "10m",
                "alertname": "HighErrorRateFastBurn",
                "exp_alerts": [
                    {"exp_labels": {"severity": "critical", "team": "checkout"}}
                ]
            }
        ]
    }

    # Write test file and run promtool
    with open("/tmp/alert_test.yaml", "w") as f:
        yaml.dump(test_case, f)

    result = subprocess.run(
        ["promtool", "test", "rules", "/tmp/alert_test.yaml"],
        capture_output=True, text=True
    )
    assert result.returncode == 0, f"Alert test failed: {result.stderr}"

Chaos-Based Alert Testing

The best way to test alerts is to trigger the conditions they detect:

1. Run a chaos experiment (kill pods, inject latency)
2. Verify the expected alert fires within the expected timeframe
3. Verify the runbook link is correct and the runbook is up to date
4. Verify the alert resolves when the chaos experiment ends

This is a natural extension of game day exercises -- include "did the right alert fire?" as a success criterion.