4.2The Chaos Engineering Cycle

Every chaos experiment follows a five-phase cycle:

  +-------------------+
  | 1. Define Steady  |
  |    State          |
  +--------+----------+
           |
           v
  +--------+----------+
  | 2. Hypothesize    |
  |    (What should   |
  |     survive?)     |
  +--------+----------+
           |
           v
  +--------+----------+
  | 3. Inject Failure |
  |    (Controlled)   |
  +--------+----------+
           |
           v
  +--------+----------+
  | 4. Observe System |
  |    Behavior       |
  +--------+----------+
           |
           v
  +--------+----------+
  | 5. Learn & Fix    |
  |    (or confirm    |
  |     resilience)   |
  +--------+----------+
           |
           +---------> Repeat

Phase 1: Define Steady State

Before breaking anything, define what "normal" looks like using quantitative metrics. This is your baseline.

Good steady state definitions:

• "Our checkout service processes 500 req/s with p99 latency under 800ms and error rate below 0.1%."
• "Order confirmation emails are sent within 30 seconds of purchase for 99.5% of orders."
• "The search service returns results in under 200ms for 95% of queries."

Bad steady state definitions:

• "The system works fine." (Not measurable)
• "CPU is under 80%." (Resource metric, not user-facing)
• "No alerts are firing." (Absence of alerts is not evidence of health)

The key distinction: steady state should be defined in terms of user-visible behavior, not infrastructure metrics.

Phase 2: Hypothesize

Form a specific hypothesis about what the system should do when the failure occurs. The hypothesis should be falsifiable.

Examples:

• "If we kill 50% of the checkout service pods, the remaining pods should handle the load with p99 latency under 2 seconds and no errors visible to users."
• "If we introduce 500ms of network latency between the order service and the payment service, orders should still complete within 5 seconds."
• "If the primary database fails over to the replica, read traffic should see no more than 5 seconds of degradation."

Phase 3: Inject Failure

Apply the fault in a controlled manner with clear boundaries:

• Scope. What components are affected?
• Duration. How long does the experiment run?
• Blast radius. How many users could be affected?
• Kill switch. How do you abort immediately if something goes wrong?

Common fault types:

Fault Category	Specific Faults	Simulates
Compute	Pod kill, node shutdown, CPU stress	Hardware failure, resource exhaustion
Network	Latency injection, packet loss, DNS failure, partition	Network degradation, cross-AZ issues
Storage	Disk fill, I/O latency, disk failure	Storage issues, noisy neighbors
Application	Process kill, memory leak, thread exhaustion	Application bugs, resource leaks
Dependency	External service timeout, rate limit, wrong response	Third-party failures
Time	Clock skew, NTP failure	Time synchronization issues

Phase 4: Observe

Monitor the system during and after the experiment. Compare actual behavior against your hypothesis. Key observations:

• Did the system stay within the steady state definition?
• How long did recovery take?
• Were users affected? How many? For how long?
• Did alerting fire correctly? Was the right team paged?
• Did auto-scaling or self-healing mechanisms activate?

Phase 5: Learn and Fix

Every experiment produces one of three outcomes:

1. Hypothesis confirmed. The system handled the failure gracefully. Document the resilience mechanism and schedule the experiment to run regularly.
2. Hypothesis disproved. The system degraded beyond acceptable limits. This is actually the most valuable outcome -- you found a weakness before your customers did. File a bug, fix it, and re-run the experiment.
3. Unexpected behavior. The system behaved in a way nobody predicted. This is common and often reveals missing observability, incorrect assumptions, or cascading failure paths.