51 / 56 · 18 Test Management Tools · AI in Test Management← prev⊞ allnext →☰ Read as one page
7.4Flaky Test Detection
AI identifies patterns in test results that humans would need hours of manual analysis to find.
Patterns AI Detects
| Pattern | Description | Likely Cause |
|---|---|---|
| Alternating pass/fail | Test oscillates without code changes | Race condition, timing dependency |
| Time-of-day correlation | Fails only during business hours | Load-dependent, shared resource |
| Runner-specific failures | Fails on runner-3 but passes elsewhere | Environment configuration issue |
| Monday morning failures | Fails after weekend, passes by Tuesday | Expired tokens, stale data, certificate renewal |
| Cascade failures | When test A fails, tests B, C, D always fail too | Test dependency, shared state |
What to Do With Flaky Tests
- Quarantine: Move the flaky test to a separate suite that does not block PRs
- Track: Label it in Jira (e.g.,
flaky-test) with root cause analysis - Fix: Address the root cause (timing, state, environment)
- Restore: Move back to the main suite after the fix is verified over multiple runs
- Monitor: Track flake rate trends to ensure overall test suite health is improving
Flake Rate Threshold
A healthy test suite has a flake rate below 2%. Above 5%, developers start ignoring test failures, and the test suite loses its value as a safety net.