Why Most AI Testing Pilots Never Scale

Almost every testing team is now "using AI." Almost none of them are using it at scale. The distance between those two sentences is where most of this year's testing budgets are quietly going to die.

In the summer of 1999, Hershey flipped the switch on a brand-new enterprise system — a huge, expensive, thoroughly demoed suite meant to modernise how the company took and shipped orders. The software itself worked. What didn't work was everything around it: a rushed go-live, jammed into the worst possible moment of the calendar, before the org had really learned to run it. The result is now a business-school ghost story. Hershey couldn't get roughly $100 million of chocolate out the door for Halloween. Kids across America got their sweets from somebody else that year, and it wasn't because the code couldn't add up an order. It was because a working tool and a working practice are not the same thing, and the company discovered the gap between them in October.

That gap is the single most important thing to understand about AI in testing right now.

The reason most AI testing pilots stall isn't the model — it's everything around the model: who owns the tool, what data it's allowed to touch, whether anyone actually trusts its output, and how it plugs into a pipeline built years before it existed. A demo needs none of that. A production practice needs all of it. Which is exactly why the demo dazzles and the rollout quietly rots.

The numbers say this is happening at scale. Per Capgemini's World Quality Report 2025-26, 89% of organisations are piloting or deploying generative AI in quality engineering — but only around 15% have scaled it enterprise-wide. Nine teams in ten are experimenting. Roughly one in seven has made it real.

Haven't we seen this exact movie before?

We have, and it's worth naming, because it stops you feeling uniquely cursed. Back in 1987, the economist Robert Solow looked at a decade of frantic corporate computerisation and delivered the driest sentence in the history of technology: "You can see the computer age everywhere but in the productivity statistics." Everyone had bought the machines. The payoff was missing. It took years — new processes, new skills, a generation of managers who actually understood the tools — before the productivity showed up. The technology had arrived long before the practice did.

AI in QA is sitting in its own Solow moment. It's everywhere in the pilot decks and nowhere in the throughput numbers, and for the same unglamorous reason: buying the capability is a purchase, but absorbing it is a slog. FoxMeyer, once the fourth-largest drug distributor in America, bet the company on a similarly ambitious rollout and the botched implementation helped push it into bankruptcy by 1996. The tools in these stories were rarely the villain. The org that couldn't metabolise them was.

Why do AI testing tools work in a demo and fail in production?

Because a demo is a controlled happy path and production is your actual, specific mess. And the things that break in the mess are boringly organisational, not technical.

Look at what teams say stops them scaling, in that same World Quality Report: data privacy (67%), integration complexity (64%), reliability and hallucination worries (60%). Half of organisations flatly said they lack the in-house AI expertise to go further. Not one of those is a prompt you can improve your way out of. They're questions about governance, plumbing, and skills — the same three questions that sank Hershey's Halloween.

The people doing the work put it less politely. In a discussion surfaced through Ranorex's community coverage, one tester described trying three different AI testing tools in a single year: "each one promised to reduce our maintenance overhead, and each one ended up creating more work than it saved." Another line from the same coverage is the kind of thing you only say after being burned: "the biggest problem with AI testing tools is that they're designed by people who don't actually do testing." (Both reach us secondhand, through a vendor's blog rather than the original thread, so hold them as sentiment rather than census — but the sentiment is not hard to find.)

Nine teams in ten are running a pilot. One in seven is running it for real. The distance between those two numbers is the entire job.

Isn't "we're using AI" the same as adoption?

No — and quietly conflating the two is how the budget evaporates without anyone deciding to spend it. Experimenting is one enthusiastic engineer, one tool, one good week. Adopting is that tool embedded in the pipeline, owned by a named human, trusted by the team, and still doing its job when that engineer is on holiday. The whole distance between them is governance, data access, and verification: the unglamorous scaffolding nobody photographs for the launch post.

The trust problem alone will strangle a rollout in its cot. Across the broader developer world, Stack Overflow's 2025 survey found 84% now use or plan to use AI tools — but only 33% trust their accuracy, and active distrust actually rose year on year. You cannot build a durable testing practice on a tool two-thirds of your people quietly don't believe. Zoom out further and Deloitte's 2026 outlook put the share of companies with AI agents genuinely operational in production at around 11%. The chasm between the excitement and the shipping is an entire industry's condition, not your team's private shame.

How do you actually cross the gap?

You stop trying to "adopt AI," which is not a goal so much as a press release, and you pick one specific, owned, verifiable workflow instead.

Start narrow enough to be almost embarrassing. "Generate the first draft of our API test data" is a real, checkable job. "Transform our QA with AI" is a slide. The narrow thing you can verify is a thing you can grow; the grand mandate is a thing you can only announce and then quietly abandon. Then give it an owner — a named person responsible for its output being right — because a tool nobody owns is just a pilot with an expiry date nobody's written down.

Settle the data question before you settle the tool question, since privacy is the number-one thing teams say blocks them. Decide what the model may touch, and never paste production data into a third-party model to make a demo look good — this shouldn't need saying, yet here we are. Build the human verification step in from day one rather than promising to add it later, because "later" is where pilots go to die; the tools that survive contact with production are the ones used to generate candidates a person then checks, never to decide. And measure the maintenance, not the magic. Every failed rollout in the research shares one symptom: the tool created more upkeep than it removed. A tool with negative net time is a tool to drop, however good it looked in the room.

The short version

89% of QE orgs are experimenting with GenAI; only ~15% have scaled it (World Quality Report 2025-26). Experimenting is not adopting.
It's a Solow productivity paradox for testing: the tools are everywhere, the payoff lags — because absorbing a capability is slower than buying it.
Pilots stall on data privacy (67%), integration (64%), reliability (60%) and a plain expertise gap — org problems, not model problems.
Only 33% of developers trust AI output (Stack Overflow 2025); you can't scale a practice the team doesn't believe in.
Cross the gap by scaling one owned, verified, narrow workflow — and drop anything that costs more maintenance than it saves.

Turning an impressive AI demo into a testing practice that survives production — narrow scope, a named owner, verification you can defend — is the actual work, and it's the work Hershey skipped. That's the core of the Pearly Quality AI-in-testing workshop: where AI genuinely earns its place in a QA process, and where it's just maintenance with a shinier name.