Why Automated Accessibility Tools Only Tell Half the Story

The audit that passed — until a real user tried it

A civic technology team ran their new portal through Axe and Lighthouse, fixed every flagged issue, and updated their accessibility statement to show compliance. Three months later, they got a complaint from a user with low vision who couldn't complete the main task flow because hover-dependent tooltips revealed critical instructions that were never surfaced any other way.

None of that showed up in the automated scan. It never would have.

Automated accessibility testing tools are genuinely useful. They find real problems quickly and consistently. But they're testing the code, not the experience. And a significant portion of what makes a website accessible or not accessible lives in the experience, not in the markup.

What automated tools actually test

Axe, WAVE, Lighthouse, and similar tools work by running a set of rule-based checks against the DOM (the rendered HTML of your page). They're looking for things that can be evaluated programmatically:

Is this image missing an alt attribute?

Is this link missing visible text?

Does this color combination fail contrast ratio thresholds?

Is this form field missing a label?

Is this landmark element present?

These are valuable checks. Many of the most common accessibility failures are detectable this way. But the rule-based approach has a hard ceiling.

Research from the accessibility community — including published methodology audits from Deque, AbilityNet, and the UK's Government Digital Service — consistently shows that automated tools detect roughly 30 to 40 percent of accessibility issues. The rest require human judgment or real user testing.

What automated tools can't test

Whether content is actually understandable

An image has an alt attribute. But is the alt text accurate and useful? Does it convey what a sighted user would understand from the image? A tool can confirm the attribute exists; it can't evaluate whether "image.jpg" or a two-sentence summary is appropriate for the context.

Reading order and logical flow

Screen readers read content in DOM order, which may or may not match the visual order. A page can be visually well-organized but present information to screen reader users in a confusing sequence because of how the HTML is structured. Automated tools can flag some ordering issues, but evaluating whether the sequence makes logical sense requires reading the content.

Cognitive load and task complexity

WCAG 2.1 introduced success criteria around cognitive accessibility, but they're difficult to test programmatically. Does each page have a clear purpose? Are error messages specific and helpful? Is the navigation structure predictable? Is there too much going on visually for users with attention or processing differences? These are judgment calls.

Focus order and keyboard flow

Tools can confirm that interactive elements are keyboard-accessible. They can't confirm that the focus order makes sense during a real task. On a complex form with conditional fields, the logical focus path may be completely different from what the DOM order suggests.

Interaction design failures

Hover-dependent interactions, time-limited content, complex component behavior (e.g., accordions, tabs, carousels), and dynamic content updates require hands-on testing to evaluate for accessibility.

What actually happens on a screen reader

Automated tools simulate what a screen reader would announce, but they don't replicate the experience. Running VoiceOver or NVDA on your own site for 15 minutes reveals a different class of issues than any automated scan will surface.

A practical manual testing routine

The goal of manual testing isn't to replace automated checks — it's to catch what automation misses. This routine takes two to three focused hours per major page template and should happen at least once before launch and after any significant content or interaction change.

Step 1: Keyboard-only navigation

Unplug your mouse (or set it aside). Navigate your entire site using only Tab, Shift+Tab, Enter, Space, and the arrow keys.

Check for:

Visible focus indicator on every interactive element at all times

Logical tab order (left to right, top to bottom, following content hierarchy)

No keyboard traps (you can always exit any component and continue navigating)

All interactive elements reachable (no items that can only be activated by mouse)

Dropdowns and modals behave predictably: they open, allow navigation within, and close with Escape

Time needed: 20-30 minutes per page template.

Step 2: Screen reader spot check

Use VoiceOver on macOS or iOS (turn on with Command+F5 or triple-click the side button), or NVDA on Windows (free download from nvaccess.org). You don't need to be an expert user. Listen for:

Images being announced with useful descriptions (not "image" or the file name)

Links making sense when read aloud in isolation

Form labels being announced correctly when you tab into each field

Headings creating a navigable outline (navigate by heading with VoiceOver using VO+Command+H or NVDA's H key)

Dynamic content updates (search results, error messages, alerts) being announced

Use the simplest navigation mode first: Tab through the page and listen to what gets announced. Then try navigating by headings to see if the page structure makes sense.

Time needed: 30-45 minutes per page template.

Step 3: Zoom and viewport testing

Zoom your browser to 200% and stay at that zoom level while navigating normal page flows. Check for:

Text that becomes cut off or hidden

Content that overlaps

Horizontal scrolling where it shouldn't appear

Loss of functionality at zoom

Then switch to a mobile viewport (browser DevTools → responsive design mode) and repeat keyboard and screen reader checks. Many accessibility failures only appear at certain viewport sizes.

Time needed: 20 minutes per page template.

Step 4: Cognitive load review

Read each page aloud as if you were encountering it for the first time. Ask:

Is the primary purpose of this page clear within the first two sentences?

Are all error messages specific (do they tell you what went wrong and how to fix it)?

Is there any content that only appears on hover or that requires remembering something from a previous step?

Are there any instructions that assume familiarity with the interface?

Is there anything visually complex that isn't explained?

Time needed: 15 minutes per page template.

Combining automated and manual testing

Here's a practical integration model for a content team:

Automated: Run on every deploy, ideally as part of a CI/CD pipeline or as a browser extension check before publishing any page. Axe DevTools as a browser extension is free and runs on any page in Chrome or Firefox. WAVE is the easiest for non-technical team members.

Manual keyboard pass: Run before any new page template goes live and before any significant interaction redesign. Assign to a developer or senior content manager.

Screen reader spot check: Run quarterly on your highest-traffic page templates and before any significant site launch or migration. Assign to someone who practices consistently enough to be fluent in basic screen reader navigation.

Real user testing: Run annually if budget allows. Recruit users with disabilities through services like Fable or UserZoom Disability Panels. There is no substitute for watching a real user navigate your site.

The compliance trap

One of the most dangerous things about automated accessibility tools is how easy they make it to claim compliance. "We ran Axe and fixed all the errors" sounds credible. It isn't sufficient.

WCAG conformance requires meeting all applicable success criteria, most of which cannot be automatically verified. Automated tools help you move faster and build a consistent baseline. They don't produce a conformance statement.

If your organization publishes an accessibility conformance report (an VPAT or ACR), it should reflect who tested it, what methods were used, and the date range of testing. Automated-only testing should be disclosed as such.

If you want to build a testing process that gives you genuine confidence rather than a clean automated report, get in touch and we can design a workflow that fits your team size and publishing cadence.