Manual vs Automated Accessibility Testing: What Each One Actually Catches
"Can't we just run a tool?" Partly. Here's the part you can, the part you can't, and why the split is the whole point.
Every team that gets serious about accessibility asks the same thing within the first week: can't we automate this? Buy a scanner, wire it into CI, get a number that goes up. It's a reasonable instinct. Testing accessibility by hand sounds slow and specialised, and automation is neither.
So here's the honest answer, before the detail: yes, automate — but automation finds only the mechanical failures, and those are the minority. The issues that actually lock people out are usually the ones no tool can judge. You automate to clear the cheap noise fast, and you keep a human for everything that requires deciding whether something is good, not just whether it's present.
A scanner checks whether the label exists. Only a person can check whether it's the right label.
What does an automated tool actually catch?
Automated checkers — axe, Lighthouse, WAVE, Pa11y, the whole family — are pattern matchers. They're excellent at things that are true or false in the markup:
- missing
altattributes - text-contrast ratios below the threshold
- form fields with no programmatic label
- missing page language or document title
- duplicate IDs, some kinds of ARIA misuse
These are real failures, and finding them in seconds for free is genuinely useful. Run a scanner. Put it in CI. There's no reason not to.
The catch is coverage, and the honest number depends on how you count. By share of WCAG success criteria, only about a third are machine-testable at all. By volume of issues found in real audits it looks rosier — Deque's automated-coverage analysis puts axe-core at 57% — but only because a handful of automatable failures, like colour contrast, turn up almost everywhere. Measured against the guidelines themselves, most of WCAG still needs a human. So a clean automated report doesn't mean "accessible." It means "none of the machine-checkable things are broken," which is a much smaller claim than it looks.
What can only a human catch?
Everything that depends on meaning, order, or experience:
- whether alt text actually describes the point of the image, or just says "image"
- whether the Tab order matches the visual order or jumps around
- whether a custom widget — dropdown, modal, date picker — announces its role and state to a screen reader, or is silent
- whether focus lands somewhere sensible when a dialog opens and closes
- whether the page is usable by someone driving it with VoiceOver or NVDA, not just technically passing
A tool will happily green-check a button labelled "click here" that leads nowhere useful. It can't read the room. You can.
So which should I use? Both — in this order
| Automated | Manual | |
|---|---|---|
| Catches | missing alt, contrast, missing labels, language/title, some ARIA | meaningful alt, focus order, custom widgets, screen-reader experience, "does this actually work" |
| Share of WCAG | ~1/3 of success criteria (higher by issue volume) | the rest (judgment criteria) |
| Speed / cost | seconds, free, runs in CI | minutes to hours, needs a skilled human |
| Best for | fast first sweep, catching regressions | deciding whether it's genuinely usable |
| Blind spot | can't judge meaning or experience | doesn't scale to every page on every commit |
The workflow that actually works: let the scanner clear the mechanical failures on every build, then spend your human time on the judgment layer — keyboard pass, screen-reader pass, custom components. Don't burn a skilled tester's afternoon hunting missing alt attributes a tool finds in a second. And don't kid yourself that the green check means you're done.
The trap
The failure mode I see most: a team automates accessibility, watches the score hit 100, and stops. They've tested a third of the problem and declared victory on all of it. Six months later a real screen-reader user can't finish checkout, and the dashboard is still green.
Automation isn't the accessibility strategy. It's the part of the strategy you get to stop doing by hand. The rest is still yours.
If you want to build the judgment half — the keyboard and screen-reader passes, reading a custom component the way assistive tech does — that's the core of the Pearly Quality accessibility workshop. We start where the tools stop.