30 / 95 · 05 Performance & Chaos Engineering · Litmus Chaos Experiments← prev⊞ allnext →☰ Read as one page
5.7Experiment Design Best Practices
- One variable at a time. Do not inject network latency and kill pods simultaneously in your first experiments. Isolate variables to understand each failure mode independently.
- Define probes before running. If you run an experiment without probes, you are just breaking things without measuring the impact.
- Start with the smallest blast radius. Kill one pod, not 50%. Add 100ms latency, not 5 seconds. Increase gradually as you build confidence.
- Document every experiment. Record the hypothesis, configuration, results, and action items. This creates institutional knowledge about your system's failure modes.
- Make experiments idempotent. Running the same experiment twice should not leave the system in a different state. Litmus handles cleanup automatically, but verify your probes are not creating side effects.