Research Guides
Choose a Website Audit Sample That Can Find Different Failures
Checking twenty nearly identical articles may reveal less about a website than checking a small set of meaningfully different pages and states. The count of tested URLs is not the same as coverage. Start an audit by asking which kinds of failure your selection could miss.
This is a general planning worksheet for authorized website research and quality review. It is not a complete accessibility evaluation method, a statistical guarantee or a description of an automated Colquhoun.shop sampling feature.
Define the question before collecting URLs
Write the question the review should answer. “Can readers find and download a report?” calls for different evidence from “Do article pages expose consistent titles and canonical links?” Define the relevant website area, access level and test environment, along with what is outside the assignment.
Keep that boundary in the final report. A public-page review cannot establish that an authenticated dashboard works. If you cannot access a relevant area, record the coverage gap rather than replacing it with more public pages and calling the site fully checked.
Use a methodology as a reference, not a badge
W3C's WCAG Evaluation Methodology organizes accessibility evaluation around scope, exploration, representative sampling, evaluation and reporting. It addresses structured and random samples and complete processes. It also explains that evaluating a subset alone does not justify a WCAG conformance claim for an entire website. W3C: WCAG Evaluation Methodology 2.0
The worksheet here borrows the planning principle of making coverage explicit. It does not substitute for that methodology or the expertise needed to apply accessibility standards. For a formal accessibility engagement, use the relevant standard and full methodology rather than treating this short article as a checklist of requirements.
Build a coverage matrix
List the distinct things your site actually contains: article templates, comparison pages, search results, downloads, forms and account views where authorized. Next, list relevant states such as populated, empty, validation error and successful completion. Add important content differences, including a long title, a large table or a multilingual page.
Choose concrete examples against that matrix. Record why each one is included. “Typical article template with a data table” is a reason; “first URL returned by the crawler” describes convenience. A random selection can supplement deliberate coverage, but document how it was selected rather than labeling hand-picked pages random.
Work through one fictional report library
Suppose a library contains 200 short articles, 12 comparison reports and a download-request form. A review of the first 25 article URLs misses both the comparison layout and the request journey.
A better initial plan might include an ordinary article, a long article, a comparison report with a wide table, a search result with matches, an empty search result and the download request from entry through confirmation. These are invented examples, not a universal sample-size prescription. Add cases based on the site's actual functions and the consequences of failure.
For the form journey, use an authorized test environment or agreed test procedure. Do not create real purchases, contact requests or customer records merely to fill a coverage cell. If a safe completion test is unavailable, mark that step untested.
Let findings change the selection
If one comparison report has a broken table layout, test another report using the same component and a relevant contrasting case. Record the expansion and its reason. Do not silently replace a failed page with an easier one, or retain only successful examples in the final sample.
Separate a confirmed local defect from a suspected shared-template defect. The shared implementation may justify broader investigation, but it does not prove that every page has the same visible outcome.
Close with coverage, not just a score
- Purpose and boundary: the question, website area and excluded access.
- Selection: page or state, category and reason for inclusion.
- Procedure: environment, safe test data and actions actually taken.
- Evidence: observations, failures and supporting records.
- Expansion: extra cases added after a finding.
- Limits: untested journeys, variants and required follow-up.
Report “These templates and states were checked using these procedures” rather than “the website passed.” A transparent coverage record helps the next reviewer extend the work without mistaking a useful sample for proof about every page.