12.8Building a Game Day Program

Quarter 1: Foundation

• Run 1 game day per month in staging
• Focus on single-service failures
• Establish the post-game review process
• Build initial runbooks based on findings

Quarter 2: Expansion

• Run 2 game days per month
• Include multi-service scenarios
• Run first production game day (low-traffic window)
• Start tracking MTTD/MTTA/MTTR trends

Quarter 3: Maturity

• Weekly game days (rotating services)
• Production game days during normal hours
• Include security incident scenarios
• Cross-team exercises (backend + frontend + mobile)

Quarter 4: Continuous

• Automated game days (scheduled chaos with human response evaluation)
• Game days integrated into on-call rotation onboarding
• Quarterly "all hands" game days with company-wide participation
• Publish a game day report to the entire organization

Interview Talking Point: "I approach performance and resilience as two sides of the same coin. Performance testing tells you how fast the system runs under load; chaos engineering tells you what happens when that load arrives during a failure. I integrate both into CI -- k6 validates our SLOs in staging, and Litmus chaos experiments verify we stay within SLO even when pods are being killed. For LLM-backed features, I add dedicated metrics like time-to-first-token and tokens-per-second, because traditional latency percentiles do not capture the user experience of a streaming AI response. The goal is not to prove the system never fails -- it is to prove that when it fails, it fails gracefully within our error budget."