Research Guides
Weight an Uneven Website Audit Sample Before Estimating Failures
A site audit can deliberately inspect more pages from a small section because defects there would otherwise be easy to miss. That is useful for finding problems. It becomes misleading when the fraction of failures in that uneven sample is presented as the fraction of pages failing across the site.
This worksheet separates a diagnostic sample from a designed estimate. The examples are fictional and do not describe a Colquhoun.shop feature.
1. Decide whether the sample finds issues or estimates prevalence
A diagnostic sample might prioritize recent template changes and high-risk pages. Publish the findings as observations from those pages. A prevalence estimate instead needs a defined population and a sampling design that supports the inference.
Statistics Canada's Survey Methods and Practices explains stratified sampling and design weights. When a simple random sample is taken separately within each group, the population and sample counts determine how much each selected unit represents. That calculation does not retroactively turn a hand-picked set into a random sample.
2. Build a frame before drawing pages
Define which URLs count as a page for the question. Freeze a dated list, remove duplicates according to a documented rule, and assign every in-scope page to exactly one group. For example, separate product pages from support pages rather than counting the same page once by template and again by topic.
Write the pass/fail criterion before selecting the sample. Record how an inaccessible or untestable page will be handled. Replacing it with any convenient page after looking at the results can change who had a chance to be included.
3. Work a fictional disproportionate sample
Suppose the frame contains 900 product pages and 100 support pages. A simple random draw without replacement selects ten from each group. In the fictional results, two product pages and eight support pages fail the same stated check.
- Raw sample: 10 failures in 20 tested pages, or 50%.
- Product group: 2 of 10 fail; this group is 90% of the frame.
- Support group: 8 of 10 fail; this group is 10% of the frame.
- Estimated frame-level rate: 0.90 × 20% + 0.10 × 80% = 26%.
Neither number is a census result. The raw 50% describes the sampled pages; the 26% point estimate gives the groups their frame proportions under the stated design. With only ten pages per group, do not present it as an exact count of all failed pages or invent an uncertainty range.
4. Keep enough information to reconstruct the draw
- Population: dated frame, eligibility and deduplication rules.
- Groups: grouping rule and in-scope page count in each group.
- Draw: selection method, selected IDs and count for each group.
- Measurement: check definition, outcome and any missing result.
- Reporting: diagnostic observations, design-based estimate, or both with different labels.
Do not reuse the worked weights for a sample taken by a different method. If important groups have unknown sizes, selection was purposive or measurements are missing, document the limitation rather than using the calculation as a repair.
5. Publish the design alongside the number
A clear note for the fiction reads: “We randomly sampled ten pages from each of two groups. Ten of the 20 tested pages failed. Weighting each group's observed rate by its share of the eligible frame gives a 26% estimate; this is not a site census.” Keep the known frame exclusions visible. The review can prioritize repairs in support pages without claiming that its raw failure fraction applies to every page on the site.