Why Your Tests Are Flaky (It's Almost Always Timing)
Flaky tests get filed under "gremlins" — intermittent, unlucky, unknowable. They're none of those things. A flaky test is a deterministic bug wearing a costume, and the costume is almost always a timing problem.
On the morning of 1 August 2012, the trading firm Knight Capital lost about $440 million in roughly 45 minutes — close to the entire company — while everyone watched, helpless, as its systems fired millions of unintended orders into the market. The cause was not chaos, though it looked exactly like chaos. New code had been deployed to seven of the firm's eight servers. The eighth kept running an old, repurposed flag that quietly woke up a piece of long-dead software, and only that one server misbehaved. The disaster looked random from the outside. It wasn't. It was a completely deterministic error whose trigger just didn't fire everywhere at once.
That is precisely what a flaky test is, minus the eight-figure hole in the balance sheet.
Every flaky test has a deterministic root cause; it only looks intermittent because the condition that triggers the failure doesn't occur on every run. Find the condition and the mystery evaporates. And most of the time the condition is the same tired one: your test checked something before the system was ready, or two operations raced and the order changed between runs. It's a waiting problem, not a bad-luck problem — and, crucially, not a bad-selector problem, which is where whole teams burn their first day of investigation.
This isn't a niche affliction anymore, either. Across more than 10 million mobile builds tracked by Bitrise, the share of teams hitting flaky tests rose from 10% in January 2022 to 26% by June 2025 (reported via SD Times and Autonoma, 2025) — a 160% jump in three years. Google's own engineering has found that roughly 16% of its tests show some flakiness, with about 1.5% of all test executions failing incorrectly. Flakiness is now the water most test suites swim in.
Isn't a flaky test just random, though?
No, and the word the industry reached for tells you people have always half-known this. A heisenbug — coined decades ago, in a nod to Heisenberg — is a bug that changes its behaviour the moment you try to observe it. You add a log line to catch it and it vanishes. You attach a debugger and it behaves perfectly. Which sounds spooky until you realise what's really happening: the log line, or the debugger, changed the timing, and the timing was the bug all along. The ghost was never supernatural. It was a race, and you accidentally slowed one side of it down.
The most infamous timing bug in software history makes the stakes unforgettable. Between 1985 and 1987, the Therac-25 radiation-therapy machine delivered massive overdoses to patients, several of them fatal. Nancy Leveson's investigation traced it, in part, to a race condition: if an experienced operator typed the setup sequence fast enough, they beat the machine's internal state-setting and the safety logic ran against stale data. A slow operator never saw it. A fast one killed someone. Same code, same inputs, different timing — the textbook definition of a flake, in the least funny context imaginable. Your flaky test and the Therac-25 are cousins. Yours just fails a build instead of a person.
What actually makes a test flaky?
A handful of causes account for the overwhelming majority, and they're rankable. An empirical study of flaky JavaScript tests (arXiv 2207.01047) broke them down roughly as follows — the precise percentages come from a JS-focused study, so read them as a strong shape rather than gospel across every stack:
| Root cause | Share | What it looks like | The real fix |
|---|---|---|---|
| Async / wait | ~45% | The test asserts before an async call, animation, or network response has finished | Wait for a condition — element ready, response received — never a fixed sleep |
| Concurrency / race | ~20% | Two operations finish in an order the test quietly assumed was fixed | Kill the shared-state assumption; make the ordering explicit |
| Test-order dependency | ~12% | Test A passes only because test B ran first and left state lying around | Isolate state; every test builds and tears down its own world |
Underneath the table there's a single pattern: the test encoded an assumption about time that holds on most runs and breaks on the rest. The reigning champion of this genre is the hardcoded sleep(2000) — a bet that two seconds will always be enough, which is true right up until the CI box is under load and it suddenly, catastrophically, isn't.
Every flaky test has a deterministic root cause. The intermittency isn't randomness — it's a trigger you haven't found yet.
Why is "just re-run it" the worst possible fix?
Because it works, which is the trap. You hit re-run, the build goes green, the ticket closes, and nothing whatsoever is learned. Do it once and it's pragmatic. Do it twice and it's a habit. Do it as a team, for a year, and you have carefully trained a whole group of intelligent people to ignore the one signal the entire test suite exists to produce. As one engineering write-up put it, "when developers start clicking re-run as a reflex instead of investigating failures, you've already lost the signal" (Autonoma, 2025). That's the real cost — not the flaky test, but the reflex it breeds.
The honest middle path, when you genuinely can't stop to investigate right now, is quarantine rather than retry. Quarantining pulls a flaky test out of the blocking suite so it stops holding everyone hostage, while keeping a visible catalogue of what's broken and waiting (thoughtbot, 2025). Done well, it buys you time to do the real work. Done badly — quarantine as the place tests are sent to be forgotten — it's just re-run with extra steps and a folder to hide the bodies in. The whole difference is whether anything ever comes back out.
So how do you fix one properly?
Reproduce the trigger before you touch the symptom. Run the test in a tight loop, run it under artificial load, run the suite in a randomised order — whatever it takes to make the failure appear on demand. A flake you can reproduce is a bug you can fix. A flake you can only re-run is a bug you've quietly agreed to keep paying rent on forever.
Then go after the usual suspect first: kill every hardcoded wait. Replace sleep(n) with an explicit wait-for-condition, and watch a startling share of your async flakiness simply evaporate, because you've removed the bet about how long something takes. If the failure depends on order, you have state leaking between tests, so make each test own its setup and teardown until it can't inherit a neighbour's mess. And if you truly must isolate a test to unblock everyone, quarantine it out loud — with an owner's name and a date attached — because a quarantine with nobody responsible for it is just a deletion you were too polite to admit to.
The short version
- Flaky tests are deterministic bugs, not randomness — the trigger simply doesn't fire on every run, exactly like Knight Capital's one rogue server.
- The share of teams with flaky tests rose from 10% to 26% between 2022 and 2025 (Bitrise/SD Times); ~16% of Google's tests flake.
- The causes rank predictably: async/wait (~45%), races (~20%), test-order (~12%) — overwhelmingly timing, not selectors (arXiv 2207.01047, a JS study).
- Re-running is the worst fix: it destroys the signal and trains the team to ignore failures. Quarantine with an owner and a date is the honest stopgap.
- Real fixes swap fixed sleeps for wait-for-condition and isolate shared state — you're removing a bet about time, not chasing a ghost.
Deciding which flaky tests to fix now, which to quarantine, and which to delete — and holding that line against a team that just wants the build green again — is test management, not luck. That judgment is the spine of the Pearly Quality test management workshop: aiming your testing effort at real risk instead of at a red pipeline.