Colquhoun.AI
← All guides

Research Guides

A Sitemap Audit That Keeps Discovery and Indexing Separate

· Colquhoun.shop editorial

A sitemap review is useful when it answers a specific question: are we giving search engines a current, consistent list of the pages we want people to find? Counting the URLs alone will not answer it. A file can load correctly while still listing retired pages or leaving out a new section.

Google describes sitemaps as a way to help discover URLs, with no assurance that every listed page will be crawled or indexed. Use that distinction to keep the audit focused on evidence you can actually collect. Google's sitemap overview

The workflow below produces a short review sheet, an exception list and a clear handoff. It is a manual research method for a site you manage, not a claim that a particular dashboard performs a complete sitemap crawl.

1. Define the pages that belong in this review

Choose a manageable scope, such as public articles, product pages or service pages. Write down the site, section, review date and person responsible for confirming the intended URLs. Keep the original sitemap file or export so later changes can be compared with the same starting point.

Find the sitemap through the site's documented publishing setup, its robots.txt references or an existing Search Console record. Do not assume every site uses the same filename. If the file is a sitemap index, record which child files you reviewed. Checking one child is not a review of the whole index.

Create a separate list of pages that should be present, using the publishing inventory and the owner's decisions. That second list matters: inspecting only the sitemap can reveal questionable entries but cannot tell you what the sitemap omitted. Compare the two lists in both directions: intended pages missing from the sitemap, and sitemap entries absent from the intended inventory.

For this workflow, treat account screens, private workspaces and unpublished previews as outside the intended public search inventory. Do not expose confidential URLs in a shared worksheet. If the purpose of a page is unclear, ask its owner before changing its visibility.

2. Check the file before diagnosing its pages

Record whether the exact sitemap URL can be retrieved and whether the returned content is the expected file. A login page or application shell is not a usable sitemap just because a request succeeded.

For XML, validate the structure and character escaping with an XML-aware parser. A URL list uses a urlset element containing url entries with loc values. The protocol requires UTF-8 encoding; optional tags do not replace those required elements. Keep parser errors separate from page-level findings. The sitemap protocol

Google's guidance calls for absolute URLs and limits a single sitemap to 50,000 URLs or 50 MB uncompressed. Larger sets need to be split. If a CMS already generates the file, fix the underlying inclusion rule or publishing record rather than maintaining an unrelated manual copy. Google's sitemap creation guidance

For a large site, state your sampling method and its limits. Reviewing ten recent articles can reveal a repeatable template problem, but it cannot establish that every URL in a 20,000-page inventory is correct.

3. Use one evidence row per URL

Copy these fields into a spreadsheet or review document:

A 200 response means the request succeeded, not that Google has indexed the content. Google can also treat an apparently successful response containing an error page as a soft 404. Inspect the response body as well as the number. Google's HTTP status guidance

Investigate conflicting signals together. Sitemap inclusion is a weaker canonical signal than redirects or canonical link annotations, and Google advises against declaring different preferred URLs through different methods. Record the intended resolution before changing several settings at once. Google's canonical guidance

If a public page is intentionally excluded from search, do not remove its noindex directive merely to clear a worksheet warning. Google must be able to crawl a resource to see that directive; blocking crawling with robots.txt can prevent this. Resolve visibility requirements with the owner first. Google's noindex guidance

4. Make lastmod traceable to a real change

An audit date and a content-modification date serve different purposes. Record today's review date in your worksheet without automatically replacing every page's lastmod value.

Google uses lastmod when it is consistently accurate and reflects a significant update to the page. Its examples include changes to main content, structured data or links, while a copyright-year change is not significant. Google ignores the priority and changefreq values. Google's XML sitemap notes

Ask what event generates the date: an editorial revision, a database update or every deployment. If the answer is uncertain, mark the evidence as missing. Since lastmod is optional, omitting an unsupported value is preferable to inventing a history.

Keep a small change record alongside the publishing process. “Added a new delivery section on November 8” is something a reviewer can verify. “Freshened for SEO” gives the next person little to check.

5. Separate fixes from unresolved questions

Fictional example: A review covers 12 article URLs. The counts and outcomes below are invented to demonstrate the method.

The summary is “12 checked, eight without a conflict in these checks, three sitemap corrections proposed and one availability investigation.” It is not “eight indexed pages.” The review has not established that.

After authorized fixes, regenerate the sitemap, retrieve the deployed file and recheck affected rows. Keep before-and-after values. For the temporary error, leave the task open until the page's intended state is confirmed; removing it solely to make a report look cleaner hides the question.

6. Verify Google's report without overstating the result

In Search Console, a successful sitemap status means Google fetched and processed the file without errors. Discovered pages are URLs parsed from it, not a count of confirmed indexed pages. The report covers sitemaps submitted through the report or API, so an absent row does not prove Google never discovered the file another way. Search Console's Sitemaps report

Use URL Inspection for important unresolved pages. Its indexed information and live test describe different states. The live test cannot predict Google's selected canonical; that field belongs to the indexed information. Record the inspection date and, where shown, Google's last crawl date so an earlier crawl is not mistaken for evidence about a change made today. Search Console's URL Inspection guide

Finish with the scope, files reviewed, specific corrections, remaining questions and next check. A useful sitemap audit leaves a reproducible account of what changed and why, without turning a clean file into a promise of rankings or traffic.

This guide describes the built-in Web Reader and Keyword Density behavior verified for this publication. Available data depends on the page response; these tools do not measure visits or search rankings.