Research Guides
Match the Same Pages Before Reporting Before-and-After Change
A website benchmark has twenty measurements before a change and twenty afterward. That does not necessarily mean it has twenty matched comparisons. If the pages, devices or test conditions differ, placing the columns beside each other can manufacture pairs that never existed.
This guide offers a pairing ledger for website research. It covers evidence preparation, not a statistical significance test, and makes no claim that Colquhoun.AI supplies paired-test software.
1. Define what makes two observations a pair
NIST's guide to paired observations describes measurements with a natural correspondence between the two samples. Its before-and-after example pairs measurements from the same object and calculates differences within those pairs.
For a website study, write down the unit being followed. It might be the same page under a specified device and testing protocol. An equal row count or the same spreadsheet position is not a matching rule.
2. Build an explicit pairing key
Record the identity and conditions required for a valid match before reviewing outcomes. Preserve the original URL, a stable page identifier where available, device category and relevant test configuration. If a page moved, document the evidence connecting its old and new identities.
Do not silently strip meaningful URL parameters to make more matches. Likewise, do not match a mobile observation to a desktop observation simply because both concern the same page. The chosen key should reflect the question the comparison is intended to answer.
3. Reconcile a fictional three-page study
Imagine three fictional pages measured before a template change. Page A loads in 2.0 seconds, B in 4.0 seconds and C in 6.0 seconds. The later run contains C at 5.5 seconds, A at 1.8 seconds and a newly added page D at 3.0 seconds.
Joining by page identity produces two complete pairs: A changes by minus 0.2 seconds and C by minus 0.5 seconds, using after minus before. B has no later measurement, and D has no earlier one. Those missing partners are not zero-second observations.
The descriptive average change for the two complete pairs is minus 0.35 seconds. Report it as a result for those two matched pages, not a site-wide improvement. A comparison of all three rows in each run answers a different question because the page set changed.
4. Keep unmatched and ambiguous records visible
Divide the reconciliation into complete pairs, before-only records, after-only records and ambiguous matches. Duplicate keys need investigation before calculating differences. If repeated tests are intended, define how repetitions correspond rather than arbitrarily choosing the first row.
Record why observations are missing when that information is known. A failed measurement, a removed page and a deliberately changed sample are different explanations. Do not fill an unknown explanation with a convenient guess.
5. Copy this pairing ledger
- Question: change being described and population of interest.
- Unit: page or other entity followed across measurements.
- Match rule: required identity and test conditions.
- Inputs: original record identifiers and measurement dates.
- Reconciliation: complete, unmatched and ambiguous counts.
- Difference: stated direction and common measurement unit.
- Exceptions: documented exclusions, missing observations and changed conditions.
- Conclusion: scope supported by the matched records.
Retain the unmatched records with the research package. A tidy paired table should not conceal how much of the original sample disappeared during preparation.
6. Separate matching from attribution
Correctly matching observations prevents one class of error. It does not prove that the website change caused the observed difference. Other conditions may have changed between runs, and a small convenience sample may not represent the site.
Use the ledger to make those limitations inspectable. If a stronger causal or statistical claim is needed, review the study design and assumptions separately rather than treating a successful join as proof.