12.4Designing Game Day Scenarios

Scenario Template

## Game Day Scenario: [Name]

### Failure Type
[Pod kill / Network partition / Dependency failure / etc.]

### Target Service
[Service name and environment]

### Hypothesis
"When [failure], the system should [expected behavior], and the team should
[expected response] within [time limit]."

### Injection Method
[Litmus experiment / Gremlin attack / Manual kubectl / etc.]

### Success Criteria
- [ ] Detection within 5 minutes
- [ ] Incident channel created within 10 minutes
- [ ] Root cause identified within 20 minutes
- [ ] Service restored within 30 minutes
- [ ] Customer impact limited to < 1% of users

### Kill Switch
[Command to abort the experiment immediately]

### Blast Radius
[Which services, users, and regions are affected]

Scenario Ideas by Maturity Level

Beginner (First 3 Game Days):

Scenario	What It Tests	Difficulty
Kill a non-critical service pod	Auto-restart, monitoring, alerting	Low
Simulate a dependency timeout	Circuit breaker behavior, fallback logic	Low
Revoke a database password	Secret rotation process, runbook accuracy	Medium

Intermediate:

Scenario	What It Tests	Difficulty
Kill 50% of a critical service's pods	Auto-scaling, load balancing, SLO resilience	Medium
Inject 2-second latency on inter-service network	Timeout configuration, retry logic, cascading failures	Medium
Simulate a full region failure	Multi-region failover, DNS routing, data consistency	High

Advanced:

Scenario	What It Tests	Difficulty
Corrupt a database table during peak traffic	Backup/restore process, data integrity checks	High
Simulate a supply chain attack (compromised dependency)	Incident response for security events, rollback speed	High
Combined failure: high traffic + partial outage + new deployment	Real-world incident complexity, team coordination	Very High