5.4Litmus Probes: Validating Behavior During Chaos

Probes are what transform chaos experiments from "break and hope" into "break and measure." Litmus supports four probe types:

HTTP Probe

Continuously checks that an HTTP endpoint returns expected responses during chaos:

probe:
  - name: "api-health-check"
    type: httpProbe
    mode: Continuous
    httpProbe/inputs:
      url: "https://api.internal/health"
      method:
        get:
          criteria: ==
          responseCode: "200"
    runProperties:
      probeTimeout: 10s
      interval: 5s
      retry: 3

Command Probe

Runs a shell command and evaluates its exit code or output:

probe:
  - name: "database-connectivity"
    type: cmdProbe
    mode: Edge    # runs at start and end of chaos
    cmdProbe/inputs:
      command: "pg_isready -h db.internal -p 5432"
      comparator:
        type: int
        criteria: ==
        value: "0"   # exit code 0 = success
    runProperties:
      probeTimeout: 15s

Prometheus Probe

Queries Prometheus and asserts metric values stay within bounds:

probe:
  - name: "error-rate-within-slo"
    type: promProbe
    mode: Continuous
    promProbe/inputs:
      endpoint: "http://prometheus.monitoring:9090"
      query: >
        sum(rate(http_requests_total{status=~"5..",service="checkout"}[5m]))
        /
        sum(rate(http_requests_total{service="checkout"}[5m]))
      comparator:
        type: float
        criteria: <
        value: "0.01"    # error rate must stay under 1%
    runProperties:
      probeTimeout: 10s
      interval: 30s

Kubernetes Probe

Checks the state of Kubernetes resources:

probe:
  - name: "min-replicas-available"
    type: k8sProbe
    mode: Continuous
    k8sProbe/inputs:
      group: apps
      version: v1
      resource: deployments
      namespace: production
      fieldSelector: "metadata.name=checkout-service"
      operation: present
    runProperties:
      probeTimeout: 10s
      interval: 10s