Colquhoun.AI
← All guides

Research Guides

Match the Same Pages Before Reporting Before-and-After Change

· Colquhoun.shop editorial

A website benchmark has twenty measurements before a change and twenty afterward. That does not necessarily mean it has twenty matched comparisons. If the pages, devices or test conditions differ, placing the columns beside each other can manufacture pairs that never existed.

This guide offers a pairing ledger for website research. It covers evidence preparation, not a statistical significance test, and makes no claim that Colquhoun.AI supplies paired-test software.

1. Define what makes two observations a pair

NIST's guide to paired observations describes measurements with a natural correspondence between the two samples. Its before-and-after example pairs measurements from the same object and calculates differences within those pairs.

For a website study, write down the unit being followed. It might be the same page under a specified device and testing protocol. An equal row count or the same spreadsheet position is not a matching rule.

2. Build an explicit pairing key

Record the identity and conditions required for a valid match before reviewing outcomes. Preserve the original URL, a stable page identifier where available, device category and relevant test configuration. If a page moved, document the evidence connecting its old and new identities.

Do not silently strip meaningful URL parameters to make more matches. Likewise, do not match a mobile observation to a desktop observation simply because both concern the same page. The chosen key should reflect the question the comparison is intended to answer.

3. Reconcile a fictional three-page study

Imagine three fictional pages measured before a template change. Page A loads in 2.0 seconds, B in 4.0 seconds and C in 6.0 seconds. The later run contains C at 5.5 seconds, A at 1.8 seconds and a newly added page D at 3.0 seconds.

Joining by page identity produces two complete pairs: A changes by minus 0.2 seconds and C by minus 0.5 seconds, using after minus before. B has no later measurement, and D has no earlier one. Those missing partners are not zero-second observations.

The descriptive average change for the two complete pairs is minus 0.35 seconds. Report it as a result for those two matched pages, not a site-wide improvement. A comparison of all three rows in each run answers a different question because the page set changed.

4. Keep unmatched and ambiguous records visible

Divide the reconciliation into complete pairs, before-only records, after-only records and ambiguous matches. Duplicate keys need investigation before calculating differences. If repeated tests are intended, define how repetitions correspond rather than arbitrarily choosing the first row.

Record why observations are missing when that information is known. A failed measurement, a removed page and a deliberately changed sample are different explanations. Do not fill an unknown explanation with a convenient guess.

5. Copy this pairing ledger

Retain the unmatched records with the research package. A tidy paired table should not conceal how much of the original sample disappeared during preparation.

6. Separate matching from attribution

Correctly matching observations prevents one class of error. It does not prove that the website change caused the observed difference. Other conditions may have changed between runs, and a small convenience sample may not represent the site.

Use the ledger to make those limitations inspectable. If a stronger causal or statistical claim is needed, review the study design and assumptions separately rather than treating a successful join as proof.

This guide describes the built-in Web Reader and Keyword Density behavior verified for this publication. Available data depends on the page response; these tools do not measure visits or search rankings.