12 / 67 · 06 Observability-Driven Testing · Canary Deployments← prev⊞ allnext →☰ Read as one page
2.6Common Canary Pitfalls
| Pitfall | Problem | Solution |
|---|---|---|
| Insufficient traffic | Cannot reach statistical significance | Increase canary percentage or observation window |
| Only checking averages | Masks tail latency regressions | Compare p95 and p99, not just mean |
| No automatic rollback | Human delay allows more users to be affected | Configure automatic rollback on metric threshold |
| Ignoring business metrics | Technically fast but functionally broken | Include conversion rate and error count in canary analysis |
| Same-version canary | Canary always passes because it is identical to baseline | Verify canary is actually running the new version |
| Cache warming effects | Canary starts slow due to cold caches | Allow a warm-up period before starting metric comparison |