Performance and Chaos Engineering

Updated Jul 2026

A modern QA architect does not just verify correctness -- they verify that systems perform under pressure and recover gracefully from failure. This chapter covers AI-driven load profiling, modern load testing tooling, chaos engineering principles, performance budgets in CI, and the unique challenges of performance-testing LLM-powered features.

Chapter Contents

1. Load Testing

• AI-Driven Profiling — Replace guesswork with production-derived traffic models
• k6 and Locust — Hands-on scripting with the two leading open-source load testing tools
• Tool Comparison — Decision matrix for choosing the right load testing tool

2. Chaos Engineering

• Principles and Cycle — The scientific method applied to system resilience
• Litmus Experiments — Writing and running chaos experiments on Kubernetes
• Chaos Tools — Chaos Monkey, Litmus, Gremlin, Chaos Mesh, and more

3. Performance Budgets

• Lighthouse CI — Enforcing frontend performance budgets automatically
• Web Vitals in CI — GitHub Actions workflows for performance gates

4. LLM Performance

• LLM Metrics — TTFT, tokens per second, cold start latency, rate limit headroom
• Load Testing LLM Endpoints — k6 scripts for AI endpoint stress testing

5. SRE Skills

• SLO, SLI, Error Budgets — The SRE framework QA architects must master
• Game Days — Designing and running incident response exercises

6. Cloud-Native Performance

• Serverless Performance — Cold starts, concurrency limits, and testing strategies
• Kubernetes Scaling — Validating HPA behavior and architecture-specific testing

Why This Matters

Performance and resilience are two sides of the same coin. Performance testing tells you how fast the system runs under load; chaos engineering tells you what happens when that load arrives during a failure. Together, they give QA architects a complete picture of system quality under real-world conditions.

Core principle: If you cannot measure it in CI, it will regress in production.