Effectively Combining Automated Tests and Manual Review

How to complement automated accessibility tests with effective manual checks?

Automated checks cover repeatable rules early and across the entire library, but cannot assess understanding and complete user paths. Therefore, manual keyboard, zoom, and assistive technology tests examine risk-based tasks and feed recurring findings back into rules and components.

Targeted risk sampling

• Percentage of relevant releases with a passed automated baseline scan and documented manual review of the affected user tasks.

• Percentage of manual barrier findings that lead to a central component fix, a new rule, or extended regression analysis.

Manual usability assessment

1. Continuously perform rule checks in development, the component library, and on the site crawl with clearly defined responsibilities.

2. Perform representative core tasks using keyboard, zoom, and selected tools according to a repeatable protocol.

3. Review manual findings for automatable causes and derive new rules, component checks, or risk sampling from them.

Green scan as approval

• Green scan as approval An automated run fails to detect any rule violations, even though focus sequencing or understandable error feedback continues to fail.

• Arbitrary manual review Manual tests are switched without documented task, thus providing neither comparability nor regression detection.

• Separate reporting pathways Insights from usage tests do not improve linting rules, components, or future sampling.

Working Example: "Green Scan as Approval"

The automated test confirms the correct names and roles of a multi-step form. However, a keyboard test reveals that after an error, the focus jumps back to the beginning without explanation; the process is corrected and supplemented with a manual regression test, while a new check will automatically report missing error relationships in the future.

Automatable Rule Range

• Targeted risk sampling Page type, novelty, usage, and known edge cases determine the manual selection instead of convenient random hits.

What to consider when combining accessibility tests

A suitable in-depth resource is available Use ARIA only where native HTML is insufficientWhen is ARIA necessary and when is a native HTML element the better solution?

In addition: Combine Automated Checks with Targeted Manual Review.

If you want to put "combining accessibility tests effectively" into practice, you can find more information at Robust Website Systems Refer back to this. The focus there is on "Component Quality, Tests, and Contrast" and "Automatable Rule Range."

Conclusion: Combining Accessibility Tests Effectively

Automation provides scope and repeatability, while manual testing assesses actual usability and comprehensibility. Quality arises from the feedback loop between both approaches.

Sources and Further Information

These primary sources make assumptions, system boundaries, and testing methods for "Combining Accessibility Tests Effectively" comprehensible.

• Understanding Non-text Contrast – W3C WAIOfficial delineation of contrast requirements for UI components, states, and meaningful graphics.

• Evaluating Web Accessibility Overview – W3C WAIOfficial W3C statement that no single tool can determine conformance and that expert human evaluation is required.

• Understanding Contrast (Minimum) – W3C WAIOfficial explanation of text contrast, its thresholds, exceptions, and test conditions.