19 / 67 · 06 Observability-Driven Testing · A/B Tests as Quality Gates← prev⊞ allnext →☰ Read as one page
3.6When A/B Testing Beats Simple Canary
| Scenario | Simple Canary | A/B Quality Gate |
|---|---|---|
| Backend API change | Sufficient (error rate + latency) | Not needed |
| UI redesign | Misses UX impact | Captures engagement metrics |
| Recommendation algorithm | Misses quality impact | Captures conversion, CTR |
| AI model update | Misses output quality | Captures quality scores, user satisfaction |
| Pricing/business logic | Misses revenue impact | Captures revenue per session |
Use simple canary analysis for infrastructure and backend changes. Use A/B quality gates when the change affects user-facing behavior where engagement and business metrics matter.