79 / 95 · 05 Performance & Chaos Engineering · Game Days: Incident Response Testing← prev⊞ allnext →☰ Read as one page
12.6Running a Game Day: Step-by-Step
T-60 minutes: Pre-brief
Gather participants in a shared call. Brief them:
- "We are running a game day exercise starting in 60 minutes."
- "This is a learning exercise, not a test. There are no wrong actions."
- "We will inject a failure in the [staging/production] environment."
- "Please respond as you would to a real incident."
- For production exercises: "Customer impact is expected to be minimal. If impact exceeds [threshold], we will abort."
T-0: Inject the failure
Execute the chaos experiment. Start the clock.
T+5 to T+30: Observe without intervening
The facilitator observes and documents but does not guide the response. The goal is to see how the team responds naturally. Only intervene if:
- The blast radius exceeds the agreed boundary
- The team is about to take a destructive action
- The exercise has gone on significantly longer than planned
T+30 to T+60: Wrap up
- Abort the chaos experiment if it has not resolved naturally
- Verify the system has returned to steady state
- Collect initial impressions from participants
T+24h to T+48h: Post-game review
Run a blameless retrospective:
- Timeline reconstruction -- What happened and when?
- What went well? -- Celebrate effective responses
- What surprised us? -- Unexpected system behavior or process gaps
- Action items -- Concrete improvements with owners and deadlines