Modern QA2026The Flakiness Problem
Log inJoin
🎯 Interview Q&AAIAPISQLPlaywrightSecurityPerformanceMobileAccessibilityCI/CDGitLinuxAgileStrategyCareer
1 / 3 · Book 13 · The State of Browser Automation in 2026 · drill: interview Q&A⊞ allnext →Get the book →

1.3The Flakiness Problem

If there is one word that defines the browser automation industry's central failure, it is "flaky."

A flaky test is a test that sometimes passes and sometimes fails without any change to the code under test or the test itself. Run it once, it passes. Run it again, it fails. Run it a third time, it passes. The result is nondeterministic.

Flakiness is not a minor inconvenience. It is an existential threat to test automation's value proposition. Here is why:

Flaky tests destroy trust. When a test fails, the team needs to know: is this a real bug, or is the test flaky? If they cannot answer that question quickly, they stop trusting the test suite. Once trust is gone, people stop looking at test results. Once people stop looking at test results, the test suite is dead --- it costs money to run and provides no value.

Flaky tests are contagious. One flaky test in a suite of 500 is annoying. Twenty flaky tests make the entire suite unreliable. Engineers start adding retry: 3 to everything. They add sleep statements "just in case." They skip intermittent failures in CI. Each workaround makes the problem worse.

Flaky tests have root causes, and most of them are architectural. The industry treats flakiness as an inevitable cost of browser automation. It is not. Most flakiness comes from three sources:

  1. Race conditions between test code and application code. Your test checks for an element before the application has rendered it. This is the most common source and is largely eliminated by Playwright's auto-waiting.

  2. Shared state between tests. Test A creates a user. Test B assumes that user exists. Test C deletes all users. Run them in order, they pass. Run them in parallel, they fail. Browser contexts solve this.

  3. External dependencies. Your test hits a real API that is slow or intermittent. Network interception and mocking solve this.

Playwright does not magically eliminate flakiness. But it provides architectural solutions to the most common causes. Selenium leaves you to solve these problems yourself, with varying degrees of success.

Pro Tip: Track your flake rate. Seriously. Run your suite 10 times without changing anything. If any test fails even once, that test is flaky. A healthy suite has a flake rate below 1%. Most Selenium suites run between 5% and 30%. If you do not know your flake rate, you do not know how much of your debugging time is wasted.