Can AI Actually Fix Your Flaky Tests?

AI that repairs flaky tests has arrived, and some of it genuinely works. The trap isn't a broken fix. It's a convincing one — a test that comes back green having quietly stopped checking the thing it was written to catch.

At 4 a.m. on 28 March 1979, in the control room of the Three Mile Island nuclear plant, a light on the panel told the operators a relief valve was closed. They trusted it. They spent hours trusting it. The valve was open the whole time, venting coolant, and the reactor core was on its way to a partial meltdown — because the light had never actually been wired to the valve. It was wired to the signal that told the valve to close. It showed the command, not the world. The operators were reading a status that had nothing behind it, and reacting as though it did.

I think about that indicator light every time a testing tool offers to fix a flaky test for me.

AI can now diagnose why a test is flaky and open a pull request to fix it — but a repair that corrects the underlying race condition and a repair that just deletes the assertion produce the exact same green build, so an automated flaky-test fix is a candidate you review, never a result you trust on sight. That's the whole thing. A green tick, like that light on the panel, tells you what was reported, not what is true. And AI is now very good at producing a green tick.

Can AI really fix a flaky test on its own?

More than the sceptics want to admit, actually. This isn't a demo that falls over the moment you look at it.

The clearest evidence comes from Uber, which built a system called FlakyGuard and turned it loose on its own notoriously large test suite. In the 2025 write-up (arXiv, 2025), the tool reproduced and fixed 47.6% of the reproducible flaky tests it was handed. Developers accepted 51.8% of the fixes it proposed. And in 57.9% of cases where it helped, it saved an engineer at least a full day of the specific misery that is chasing an intermittent failure. Those aren't slide numbers from a vendor booth. That's a real deployment, at a company whose CI pain is legendary, and the results are genuinely good.

But sit with the shape of them for a second. Roughly half. Accepted by a human. Through a pull request. Even Uber's own system, at Uber's own scale, doesn't quietly reach into the branch and fix things while everyone sleeps. And the commercial tools have landed in precisely the same posture, on purpose. Atlassian's Bitbucket flaky-test agent, launched in 2026, will diagnose a flake, write a fix, and open a pull request — a draft one. It never commits to your branch. Read that design choice as the confession it is: the people who built the tool wired a human into the loop because they know what their tool gets wrong.

There's one number I'd ask you to distrust, and it's instructive that it's the shiniest one. A "4.42 out of 5" developer rating for AI-generated flaky-test patches gets quoted all over the vendor and blog ecosystem. I went looking for its source and couldn't find it in the primary research. So treat it as marketing until someone shows you the study — and notice that the habit of asking "where did that number actually come from" is the same instinct this entire topic runs on.

Why don't experienced testers trust a "healed" test?

Because they've already imagined the failure mode, and it keeps them up at night. If you want to watch the profession think this through in real time, read the Ministry of Testing discussion on AI self-healing tests. It's not a flame war. It's a room full of people who fix flakes for a living, quietly refusing to be sold.

The sharpest line in it belongs to Bill Matthews, and it dismantles the whole pitch in two sentences: "If a test is 'healed', how do I know this hasn't changed the intent of my test. If I have to go and check, then I might as well have fixed the test myself." That's the entire economic argument against unattended self-healing, delivered without raising its voice. The tool's promise is to save you the work of understanding the failure. But a fix you can't trust is a fix you have to check, and checking is the work.

Another tester, Mark Cole, named the darker risk: these tools "sound like snake oil that may result in false positives — but even worse — false negatives without any transparency." That word, transparency, is the whole game. A test's only job is to fail when the thing it guards breaks. "Heal" it by widening an assertion, stretching a timeout, or skipping the path that keeps flaking, and you might have solved the flake by amputating the test's ability to notice anything at all — masking, as Conn McKay pointed out in the same thread, whatever real bug was underneath.

That healed test isn't neutral. It's the smoke detector someone silenced at 3 a.m. by pulling the battery, because it kept chirping. Still screwed to the ceiling. Still, technically, present. Still green on the little status LED. Deaf.

A flaky-test fix and a flaky-test mute look identical from the outside. Both turn the build green. Only one of them still tests your code.

So when is it safe to accept an AI-generated fix?

The mistake is treating the pull request as a verdict. It isn't. It's a suggestion from a fast, confident junior who has never once said "I'm not sure" — and you review it exactly the way you'd review that person's work, with affection and suspicion in equal measure.

The first move is the Three Mile Island move: read the diff, not the light. The build going green tells you nothing you didn't already fear. Open the change and look at what it actually did. If the "fix" widened an assertion, extended a timeout, or slipped in a retry, your alarm should go off, because those are the moves that silence a flake and repair a flake and the two are indistinguishable at the status level.

Then make the tool — or yourself — say the root cause out loud, in one sentence. A trustworthy fix comes with a diagnosis you can repeat to a colleague: "the test asserted before the async call had resolved." If neither you nor the machine can finish that sentence, nothing has been fixed. Something has merely been disturbed until it went quiet.

After that, ask the oldest question in testing of the new version of the test: what would have to break for this to fail? If the honest answer got weaker after the fix than before it, the fix made the test worse while making the dashboard better, which is the worst trade in the building.

And when you choose between the two kinds of repair, prefer the one that removes the uncertainty over the one that hides it. Swapping a hardcoded sleep(2000) for a proper wait-for-condition is a real fix; it deletes the timing bet entirely. Wrapping the test in retry(3) is not a fix, it's a decision to stop looking, formalised in code. Sometimes an honest quarantine-and-investigate is the right call. Silent retry almost never is.

Finally — and this is the one every vendor already agreed with you on — keep a human on the merge button. Bitbucket ships a draft PR for a reason. Don't be the team that automates away the single step the tool's own designers refused to remove.

The short version

AI flaky-test fixers are real and land roughly half their fixes — useful, not magic (Uber's FlakyGuard, 2025).
The mature tools diagnose and open a pull request; a human still merges. That design is a warning, not a limitation.
The dangerous failure is a "fix" that silences the test instead of repairing it — a false negative wearing a green tick, like a valve light showing "closed" over an open valve.
Before accepting any fix: read the diff not the status, name the root cause in one sentence, and confirm the test still fails when the code breaks.
Distrust the shiniest number in the room — the "4.42/5" rating is quoted everywhere and, as far as I can tell, sourced nowhere.

Letting AI speed up your flaky-test triage without hollowing out your suite is a practice, not a toggle — knowing which fixes to accept and which to send back is the skill. That judgment is exactly what the Pearly Quality AI-in-testing workshop is built to train.