Colquhoun.AI
← All guides

Research Guides

Review URL Cleanup Before Merging Source Rows

· Colquhoun.shop editorial

A spreadsheet can make a source list look cleaner by converting every address to lowercase. It can also silently merge references that should remain separate. URL cleanup deserves its own review rule instead of borrowing the rule used for ordinary keyword text.

This worksheet is for researchers preparing a citation inventory or website audit. It describes a manual data-quality check, not an automated feature or verified capability of Colquhoun.shop.

1. Treat an address as components, not a sentence

RFC 3986 distinguishes URI components when defining case normalization: the scheme and host are case-insensitive, while other generic components are assumed case-sensitive unless defined otherwise. It also distinguishes syntax-based normalization from equivalence established through protocol behavior. RFC 3986: URI normalization

That distinction rules out a blanket “lowercase the whole column” policy. Before transforming a source list, ask which component the proposed rule changes and what evidence supports that transformation.

2. Use a deliberately awkward test pair

In a fictional inventory, one row references https://example.com/Research/Methods and another references https://example.com/research/methods. A whole-string lowercase operation produces the same value for both. That collision is a result of the cleanup rule, not proof that the original references identify the same resource.

Keep both original rows while investigating. Record whether a site-specific check establishes a relationship, rather than assuming the spelling difference is merely cosmetic. These example addresses illustrate the test; they are not claims about live pages.

Include one pair that differs only in hostname capitalization as a contrasting fixture. A useful test set needs cases the rule should combine and cases it must not combine automatically.

3. Keep the original reference beside the grouping key

Create separate columns for the original address, proposed comparison key and decision. Never overwrite the only copy of the citation with its cleaned version. A reviewer should be able to reverse a grouping decision without reopening every source.

Assign a row identifier before processing. When multiple rows map to one key, retain their identifiers in a collision report. Do not discard the second row just because a spreadsheet labels it a duplicate.

Review collisions alongside the associated claims. Two rows may refer to the same page but support different statements or point to different sections. A page-level inventory and a citation-level inventory need different grouping decisions.

4. Document each transformation

Give query-string removal, fragment removal and trailing-slash changes their own review entries instead of bundling them into an unnamed “clean URL” step. This makes disagreement visible: one transformation might suit a page count while another damages the precision of a citation.

5. Check what changed before accepting the export

Compare the number of input rows, distinct original addresses, proposed groups and unresolved collisions. Those numbers answer different questions. A shorter list is not automatically a more accurate list.

Spot-check the retained citation against its claim after grouping. If the export will be shared, exclude private or credential-bearing addresses from the public artifact rather than treating normalization as a privacy safeguard.

The acceptance test is simple: another researcher can recover the original reference, explain every merge and identify unresolved cases. Report that narrow outcome, not a guarantee that every address in the inventory is equivalent or that search engines will consolidate them.

This guide describes the built-in Web Reader and Keyword Density behavior verified for this publication. Available data depends on the page response; these tools do not measure visits or search rankings.