Research Guides
Preserve Test Order Before Averaging Website Measurements
A website test summary reports one average for twenty measurements. That number can hide an important detail: perhaps the first ten runs behaved differently from the last ten. Preserve collection order before deciding that the observations describe one stable process.
This worksheet uses a run-sequence view to review repeated website measurements. It is an exploratory method, not a Colquhoun.shop monitoring feature or a substitute for a controlled performance experiment.
1. Keep the run index with the value
NIST describes a run-sequence plot as a measured value plotted against its observation index. It can help reveal shifts in location or variation and unusual observations. NIST: Run-Sequence Plot
Assign an identifier as each measurement is collected. Keep the actual timestamp separately. Run number 10 means the tenth observation, not necessarily ten minutes after the first. Uneven intervals should not disappear behind equally spaced index labels.
2. Define what counts as a comparable run
Record the metric, page, browser, device and test steps. Include relevant conditions such as cache state and whether the same interaction was performed. A sequence assembled from different tasks needs those differences explained before its shape can be interpreted.
Retain failed or missing attempts in the collection record. They may not supply a numeric point, but removing their identifiers can make the sequence look uninterrupted. Distinguish a missing observation from a measured zero.
3. Investigate a shift without naming its cause too soon
In a fictional test, ten initial runs cluster near 400 milliseconds and ten later runs cluster near 800 milliseconds. The overall average is near 600 milliseconds, but that summary alone does not communicate the two different portions of the sequence.
The reviewer checks the run notes and finds that the test setup changed between runs ten and eleven. That is a reason to investigate comparability, not immediate proof that the setup change caused the difference.
The report preserves both portions, documents the change and proposes a controlled follow-up with consistent settings. It does not erase the earlier runs or describe the combined average as the site's universal performance.
4. Do not sort away the question
A value-sorted view can help locate the largest measurements, but it answers a different question. Once sorted by magnitude, the display no longer shows the original collection sequence.
Keep a separate exploratory view if sorting is useful. Retain the run identifiers so a reviewer can return from an unusual value to its original position and notes. A convenient spreadsheet sort should not silently change what the horizontal axis represents.
5. Copy this sequence-review card
- Measurement: exact metric, unit and collection method.
- Run: original identifier, order and timestamp.
- Task: page and interaction sequence.
- Conditions: relevant environment and recorded changes.
- Gaps: failed or unavailable observations.
- Pattern: visible shifts or unusual values.
- Follow-up: one check that could clarify the pattern.
Annotate known events using the collection record, not a story invented after seeing the chart. Leave unexplained changes explicitly unexplained until more evidence is available.
A useful conclusion states what the sequence shows, which conditions were recorded and what remains uncertain. The chart helps decide whether a single summary is adequate; it does not independently establish a deployment effect, visitor experience or search-ranking outcome.