4.3The Five Principles of Chaos Engineering

1. Start with a Steady State Hypothesis

Every experiment must define what "normal" looks like before injecting failure. Without a measurable baseline, you cannot determine whether the experiment passed or failed.

2. Vary Real-World Events

Simulate things that actually happen in production, not theoretical failures. Prioritize by probability and impact:

Priority	Event	Probability	Impact
P0	Dependency timeout (external API)	Very high	High
P0	Single pod/instance failure	High	Low-Medium
P1	Network latency between services	High	Medium
P1	DNS resolution failure	Medium	High
P2	Full AZ (availability zone) failure	Low	Very high
P2	Clock skew	Low	Medium
P3	Simultaneous multi-component failure	Very low	Catastrophic

3. Run Experiments in Production

Staging environments lie. They have different data volumes, different traffic patterns, different network topologies, and often different configurations. The only way to truly validate resilience is to test in production with safeguards.

The graduation path:

1. Start in development (validate the experiment works)
2. Run in staging (validate the system's response)
3. Run in production during low-traffic hours (validate with real infrastructure)
4. Run in production during normal hours (validate under real load)
5. Run in production continuously (prove ongoing resilience)

4. Automate Experiments to Run Continuously

A one-time chaos test proves resilience at a point in time. The system changes every day -- new deployments, configuration changes, infrastructure updates. Continuous chaos proves resilience remains as the system evolves.

# Example: CronJob for weekly chaos experiment
apiVersion: batch/v1
kind: CronJob
metadata:
  name: weekly-pod-kill-chaos
spec:
  schedule: "0 10 * * 3"  # Every Wednesday at 10 AM
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: chaos-runner
            image: litmuschaos/litmus-checker:latest
            command: ["./run-experiment", "--config", "/etc/chaos/pod-kill.yaml"]

5. Minimize Blast Radius

Use feature flags, traffic splitting, and automated rollback to limit the impact of chaos experiments:

• Start small. Kill one pod before killing 50%.
• Use canary traffic. Route only internal or test traffic to the affected component.
• Set automatic abort conditions. If error rate exceeds 5%, abort the experiment immediately.
• Run during low-traffic windows until you have confidence in the experiment design.
• Always have a kill switch. Every experiment must be stoppable in seconds.