Colquhoun.AI
← All guides

Research Guides

Weight an Uneven Website Audit Sample Before Estimating Failures

· Colquhoun.shop editorial

A site audit can deliberately inspect more pages from a small section because defects there would otherwise be easy to miss. That is useful for finding problems. It becomes misleading when the fraction of failures in that uneven sample is presented as the fraction of pages failing across the site.

This worksheet separates a diagnostic sample from a designed estimate. The examples are fictional and do not describe a Colquhoun.shop feature.

1. Decide whether the sample finds issues or estimates prevalence

A diagnostic sample might prioritize recent template changes and high-risk pages. Publish the findings as observations from those pages. A prevalence estimate instead needs a defined population and a sampling design that supports the inference.

Statistics Canada's Survey Methods and Practices explains stratified sampling and design weights. When a simple random sample is taken separately within each group, the population and sample counts determine how much each selected unit represents. That calculation does not retroactively turn a hand-picked set into a random sample.

2. Build a frame before drawing pages

Define which URLs count as a page for the question. Freeze a dated list, remove duplicates according to a documented rule, and assign every in-scope page to exactly one group. For example, separate product pages from support pages rather than counting the same page once by template and again by topic.

Write the pass/fail criterion before selecting the sample. Record how an inaccessible or untestable page will be handled. Replacing it with any convenient page after looking at the results can change who had a chance to be included.

3. Work a fictional disproportionate sample

Suppose the frame contains 900 product pages and 100 support pages. A simple random draw without replacement selects ten from each group. In the fictional results, two product pages and eight support pages fail the same stated check.

Neither number is a census result. The raw 50% describes the sampled pages; the 26% point estimate gives the groups their frame proportions under the stated design. With only ten pages per group, do not present it as an exact count of all failed pages or invent an uncertainty range.

4. Keep enough information to reconstruct the draw

Do not reuse the worked weights for a sample taken by a different method. If important groups have unknown sizes, selection was purposive or measurements are missing, document the limitation rather than using the calculation as a repair.

5. Publish the design alongside the number

A clear note for the fiction reads: “We randomly sampled ten pages from each of two groups. Ten of the 20 tested pages failed. Weighting each group's observed rate by its share of the eligible frame gives a 26% estimate; this is not a site census.” Keep the known frame exclusions visible. The review can prioritize repairs in support pages without claiming that its raw failure fraction applies to every page on the site.

This guide describes the built-in Web Reader and Keyword Density behavior verified for this publication. Available data depends on the page response; these tools do not measure visits or search rankings.