The first time a test fails for no reason anyone can find, someone investigates. They open the logs, check the diff, maybe ping the person who touched that part of the code last. Twenty minutes later, nothing. The flow works fine when you try it by hand. So someone shrugs, hits re-run, and it passes. Onward.
The fifth time the same test does this, nobody investigates anymore. Someone just re-runs it. That's not a process failure. That's a team that learned something true about that test: it fails sometimes for reasons that don't matter, so the fastest path back to work is to ignore the failure and try again.
The problem is that the test didn't get fixed. The team got trained.
What a flaky test actually costs
If you ask an engineering team how much a flaky test costs them, most will describe the obvious version: the re-run time, the CI minutes, the few minutes someone loses clicking "rerun" and waiting again. That's real, but it's the smallest part of the bill.
The larger cost is what happens to the meaning of a failure. A test suite is only useful if "red" reliably means "something is wrong." The first time a test fails flakily and turns out to be nothing, that mapping weakens a little. By the tenth time, for that specific test, it's gone.
Red doesn't mean "something is wrong" anymore. It means "probably nothing, try again." And once a team has learned that for one test, the habit doesn't stay contained to that test. It generalizes, quietly, to how people read the whole suite.
This isn't a niche problem, at least not in the environments where it's been measured carefully. According to Bitrise's Mobile Insights 2025 report, based on more than ten million mobile builds tracked between January 2022 and June 2025, the share of teams experiencing test flakiness grew from 10 percent to 26 percent over that period, more than doubling. The report is specific to mobile engineering and DevOps environments, not software teams broadly, but the detail worth sitting with is this: over the same stretch, the report put the growth in pipeline complexity at 23 percent. At least within the report's dataset, flakiness grew faster than pipeline complexity, the thing usually blamed for it.
We've watched this happen on teams that otherwise care a lot about quality. Nobody decides to start ignoring failures. It happens one re-run at a time, each one individually reasonable, until re-running has become the default first move whenever anything goes red, before anyone's looked at what actually failed.
The habit is rational, which is what makes it dangerous
It's worth being fair to the people doing this, because retry-first isn't laziness. Given a test that's flaked before, re-running it is often the correct call under time pressure. It's usually faster than investigating, it's usually right, and being right most of the time is exactly what makes the habit so sticky.
The trouble is that "usually right" is doing a lot of quiet work in that sentence. A team that's learned to re-run first doesn't have a reliable way to tell the difference between the flaky failure they've seen forty times and the one time it's flaky-looking but actually real, an intermittent failure that happens to correlate with an actual bug that only shows up under certain timing or load conditions.
Those look identical from the outside. A re-run either makes them both disappear, or it makes the real one disappear just long enough to ship.
That's the actual risk. Not that flaky tests waste time, though they do, but that they erode a team's ability to tell a real signal from noise, right at the moment a real signal would matter most.
The cost shows up exactly when you can least afford it.
Why this gets worse, not better, over time
A new test suite is usually pretty trustworthy. Most of its failures mean something, because most of the tests are new enough that they haven't accumulated the small timing issues, race conditions, and environment quirks that produce flakiness in the first place.
Flakiness tends to accumulate rather than appear all at once. A test that waits a fixed amount of time for a page to load works fine until the page gets slightly heavier. A test that depends on the order other tests ran in works fine until someone parallelizes the suite. Each of these is a small, reasonable cause, and none of them gets fixed immediately, because fixing flakiness properly takes real investigation, and a team under deadline pressure has a faster option sitting right there: re-run it.
So the flaky tests pile up, slowly, and the retry habit grows to match them. Eventually a meaningful fraction of a suite's failures are flaky by reputation, the team knows which ones, and "just re-run it" has become load-bearing infrastructure that nobody planned for and nobody owns.
Why this is hard to fix by working harder
The obvious response, go find every flaky test and fix the underlying cause, is correct and also rarely happens at the pace flakiness accumulates. Properly fixing a flaky test usually means understanding why it's flaky, which often means understanding a timing dependency or an environment quirk that's genuinely tricky to pin down. That's real, focused work, and it competes directly with writing new features and new test coverage, which usually wins, because a flaky test that gets re-run still eventually goes green, and a team under pressure optimizes for green.
This is part of why flaky-test backlogs tend to grow rather than shrink on their own. The cost of fixing one is concentrated and immediate. The cost of not fixing it is diffuse and deferred; it shows up as slightly eroded trust in the suite, spread across every release after this one, which is exactly the kind of cost that's easy to keep deferring.
What actually breaks the pattern
The pattern doesn't break by asking people to investigate more diligently. It breaks by removing the source of the flakiness that makes retry-first the rational move in the first place.
A large share of flaky failures in traditional E2E suites trace back to a familiar root cause: brittle dependencies on timing and implementation details rather than actual behavior. A test that waits a fixed number of seconds for an element to appear is going to be flaky on a slow network and fine on a fast one, regardless of whether the feature works.
A test built around describing the outcome, rather than a fixed sequence of waits and selectors, and executed by something that can actually perceive whether the page has reached the expected state, doesn't have that specific failure mode in the first place. It's not retrying past the flakiness. The flakiness was never built into how it checks.
That distinction matters more than it sounds like it should, because it changes what a red result means again. If a test that almost never flakes goes red, the rational response reverts to what it should have been the whole time: go look. The team doesn't have to relearn discipline. The thing that made discipline expensive stops being there.
The actual signal worth protecting
A test suite's real value isn't the number of tests in it. It's whether a red result still means something, on the day a release actually depends on it. That's the thing flaky tests spend down, quietly, one reasonable re-run at a time, and it's also the thing worth being deliberate about protecting, because a team rarely notices it's gone until the one time a re-run made a real bug disappear instead of a fake one.
The cost of a flaky test was never really the test. It's the muscle memory it leaves behind, long after that specific test gets fixed or deleted.
Operio is built around helping teams restore trust in their test signal by reducing the kinds of failures that teach engineers to ignore red builds in the first place.
Data: Bitrise Mobile Insights 2025
Operio Team
Product & QA
