AI & Testing

How to Review a Test the AI Wrote

Author: imola

AI will write you a hundred tests before lunch, every one of them green. That's the problem, not the achievement. A green test that can't fail isn't protection — it's a reassuring light wired to nothing.

In 2015, Volkswagen got caught running one of the most elegant cheats in engineering history. Its diesel cars could tell when they were being emissions-tested — the telltale pattern of the wheels turning but the steering never moving, for a set duration — and in those moments they quietly switched to a cleaner-running mode. In the lab, the cars passed beautifully. On the road, freed from the test, some of them emitted up to forty times the legal limit of nitrogen oxides. The tests weren't wrong, exactly. They ran, they went green, they certified the car. They just certified a performance staged for their benefit, not the thing they were meant to measure.

I've come to think of a badly-reviewed AI-generated test as a defeat device you installed on yourself.

A test is worth keeping only if it can fail when the code breaks — and AI is extremely good at producing tests that look thorough, assert plenty, and would pass no matter what the code actually did. Reviewing an AI-generated test isn't about style, or coverage, or how many assertions it has. It's about one question: would this notice if the behaviour it's supposed to guard quietly went wrong?

The philosopher Karl Popper got to the bottom of this decades before anyone was vibe-coding unit tests. A claim that no possible observation could ever contradict, he argued, isn't a strong scientific theory — it's not a theory at all. "Irrefutability," he wrote, "is not a virtue but a vice." A test that cannot fail is the software version of exactly that vice: it forbids nothing, so it tells you nothing, while wearing the green badge of something that does.

Don't AI-generated tests have more assertions, not fewer?

They do, and that's the trap rather than the reassurance. A 2026 empirical study found AI-generated tests carried a median of 2.00 assertions per method against 1.00 for the human-written ones (arXiv, and the difference was statistically significant at p<0.001). More assertions. Sounds like more rigour. But the same study was careful to say it did not measure mutation score or fault detection — whether any of those assertions would actually catch a bug — and it flagged the AI tests for "assertion roulette," the smell of piling on assertions where not one of them clearly checks the thing that matters.

That single data point is the whole misunderstanding in miniature. Assertion count is not bug-catching power. A test can bristle with assertions and still verify nothing that would ever break. The example that does the rounds, for good reason, is expect(result).toBeDefined(). It passes when result is 90. It passes when result is 0. It passes when it's -5, and it passes when it's the string "banana" (qaskills.sh, 2026). Four wildly different behaviours, one serene green tick. The assertion is present, confident, and completely empty — Popper's unfalsifiable claim, rendered in Jest.

A test that can't fail isn't a weak test. It's a decoration that lies to you every time it passes.

What does a bad AI-generated test actually look like?

It has a handful of tells, and once you've seen them you can't stop seeing them. There's the tautological assertion — toBeDefined(), not.toBeNull(), the immortal expect(true).toBe(true) — technically present, functionally meaningless. There's mirror logic, where the test re-implements the production code to compute its own expected value, so it runs your logic twice and then congratulates itself for agreeing; it can never disagree with the code, which was the only useful thing it could have done. There's over-mocking, where so much of the thing under test is mocked out that the test is really just grading the mocks, green and proud and testing nothing real.

And there's the sheer genericness, which a QA engineer on r/QualityAssurance captured perfectly: "you still need to provide enough context or it will spit out something so generic it's worthless" (u/ferndave). Starved of your product's real risks, the model writes the average test for the average feature — plausible, tidy, and beside the point.

None of this is hypothetical. Over on r/ExperiencedDevs, a developer described the modern version of the nightmare in one exhausted sentence: "just yesterday I was asked to take over a PR that was 100% vibe coded by upper management. 150+ new files trying to be added and a whole laundry list of issues" (u/endurbro420, 280 upvotes). Or, as someone on Hacker News put it with real economy, "AI has become very good at writing pointless and bad tests, at least" (u/9rx). The tests arrive by the hundred. The reviewing does not scale to meet them, which is exactly why you need a fast, ruthless way to review each one.

So how do you actually review one?

Run every AI-written test through the same short, brutal gate. Start with the Popper question: what would have to break for this to fail? If you can't answer, or the answer is "almost nothing," delete it — you've found an unfalsifiable test, and it's worse than no test because it radiates false confidence.

Then do the thing the VW inspectors eventually did: stop trusting the staged result and break the code on purpose. Change a + to a -, flip a boolean, return the wrong value, and watch whether the test goes red. A test that stays green while you actively sabotage the code it covers is a defeat device. This is mutation testing done by hand, and tools like Stryker, mutmut, or PIT automate it across a whole suite — deliberately introducing small bugs to see which ones your tests fail to catch. It's the community's standard weapon against the perpetually-green, and it measures the exact thing coverage only pretends to.

While you're in there, check what's been mocked — if the subject itself is mostly mocked, unmock it, because right now you're testing the scaffolding. And look for mirror logic; if the expected value is computed the same way the code computes it, hard-code the expected result instead so the test can actually disagree. Finally, feed the model your real risks. AI matches the quality of the examples it's given: point it at good seed tests and your genuine failure modes and it improves markedly; drop it on a greenfield with no context and it writes confident nothing.

Because that last point cuts both ways, here's the fair version, and I mean it: this isn't all doom. Developers repeatedly report AI tests getting genuinely good with solid example files and an explicit "don't write tautological assertions" instruction. The tool is not the enemy here. The unreviewed green tick is.

The short version

  • A test earns its place only if it can fail when the code breaks — Popper's falsifiability, ported to CI.
  • AI tests carry more assertions (median 2.00 vs 1.00 human, arXiv 2026), but that study didn't measure fault detection — density is not bug-catching.
  • Watch for the signatures: tautological assertions, mirror logic, over-mocking, and generic-to-the-point-of-useless.
  • Review by breaking the code and confirming the test goes red — mutation testing by hand, or automated with Stryker, mutmut, PIT. A test that stays green while you sabotage the code is a defeat device.
  • AI tests improve sharply with real context and seed examples — the failure mode is the unreviewed green tick, not the tool.

Using AI to draft tests without quietly hollowing out your safety net comes down to a single skill: telling a test that protects you from a test that merely looks like it does. That's exactly what the Pearly Quality AI-in-testing workshop is built to train — keeping the speed of generation without losing the one thing tests are actually for.