Research Guides
A Sitemap Audit That Keeps Discovery and Indexing Separate
A sitemap review is useful when it answers a specific question: are we giving search engines a current, consistent list of the pages we want people to find? Counting the URLs alone will not answer it. A file can load correctly while still listing retired pages or leaving out a new section.
Google describes sitemaps as a way to help discover URLs, with no assurance that every listed page will be crawled or indexed. Use that distinction to keep the audit focused on evidence you can actually collect. Google's sitemap overview
The workflow below produces a short review sheet, an exception list and a clear handoff. It is a manual research method for a site you manage, not a claim that a particular dashboard performs a complete sitemap crawl.
1. Define the pages that belong in this review
Choose a manageable scope, such as public articles, product pages or service pages. Write down the site, section, review date and person responsible for confirming the intended URLs. Keep the original sitemap file or export so later changes can be compared with the same starting point.
Find the sitemap through the site's documented publishing setup, its robots.txt references or an existing Search Console record. Do not assume every site uses the same filename. If the file is a sitemap index, record which child files you reviewed. Checking one child is not a review of the whole index.
Create a separate list of pages that should be present, using the publishing inventory and the owner's decisions. That second list matters: inspecting only the sitemap can reveal questionable entries but cannot tell you what the sitemap omitted. Compare the two lists in both directions: intended pages missing from the sitemap, and sitemap entries absent from the intended inventory.
For this workflow, treat account screens, private workspaces and unpublished previews as outside the intended public search inventory. Do not expose confidential URLs in a shared worksheet. If the purpose of a page is unclear, ask its owner before changing its visibility.
2. Check the file before diagnosing its pages
Record whether the exact sitemap URL can be retrieved and whether the returned content is the expected file. A login page or application shell is not a usable sitemap just because a request succeeded.
For XML, validate the structure and character escaping with an XML-aware parser. A URL list uses a urlset element containing url entries with loc values. The protocol requires UTF-8 encoding; optional tags do not replace those required elements. Keep parser errors separate from page-level findings. The sitemap protocol
Google's guidance calls for absolute URLs and limits a single sitemap to 50,000 URLs or 50 MB uncompressed. Larger sets need to be split. If a CMS already generates the file, fix the underlying inclusion rule or publishing record rather than maintaining an unrelated manual copy. Google's sitemap creation guidance
For a large site, state your sampling method and its limits. Reviewing ten recent articles can reveal a repeatable template problem, but it cannot establish that every URL in a 20,000-page inventory is correct.
3. Use one evidence row per URL
Copy these fields into a spreadsheet or review document:
- Listed URL: Preserve the exact sitemap entry, including its host and path.
- Source file: Name the child sitemap and record when you retrieved it.
- Intended public page: Record the destination the content owner expects people to find.
- Request result: Note the initial response, any redirects and the final URL.
- Page content: Confirm that the final page contains the expected material rather than an error or login prompt.
- Search directives: Record relevant robots rules and any noindex directive in the page or HTTP headers.
- Canonical evidence: Keep the page-declared canonical separate from Google's selected canonical, if available.
- Modification evidence: Copy lastmod if present and record the content change that supports it.
- Decision: Keep, investigate, correct the generator or remove the listing, with an owner and a verification step.
A 200 response means the request succeeded, not that Google has indexed the content. Google can also treat an apparently successful response containing an error page as a soft 404. Inspect the response body as well as the number. Google's HTTP status guidance
Investigate conflicting signals together. Sitemap inclusion is a weaker canonical signal than redirects or canonical link annotations, and Google advises against declaring different preferred URLs through different methods. Record the intended resolution before changing several settings at once. Google's canonical guidance
If a public page is intentionally excluded from search, do not remove its noindex directive merely to clear a worksheet warning. Google must be able to crawl a resource to see that directive; blocking crawling with robots.txt can prevent this. Resolve visibility requirements with the owner first. Google's noindex guidance
4. Make lastmod traceable to a real change
An audit date and a content-modification date serve different purposes. Record today's review date in your worksheet without automatically replacing every page's lastmod value.
Google uses lastmod when it is consistently accurate and reflects a significant update to the page. Its examples include changes to main content, structured data or links, while a copyright-year change is not significant. Google ignores the priority and changefreq values. Google's XML sitemap notes
Ask what event generates the date: an editorial revision, a database update or every deployment. If the answer is uncertain, mark the evidence as missing. Since lastmod is optional, omitting an unsupported value is preferable to inventing a history.
Keep a small change record alongside the publishing process. “Added a new delivery section on November 8” is something a reviewer can verify. “Freshened for SEO” gives the next person little to check.
5. Separate fixes from unresolved questions
Fictional example: A review covers 12 article URLs. The counts and outcomes below are invented to demonstrate the method.
- Eight URLs load their intended articles and have consistent recorded signals.
- Two listed addresses redirect to replacement articles. Confirm the replacements are the intended preferred URLs, then update the sitemap generator to list them directly.
- One URL serves an intentionally noindex preview. Keep the preview's visibility setting and remove it from the intended public sitemap inventory.
- One URL returns a temporary server error. Investigate availability before deciding that the content should be retired.
The summary is “12 checked, eight without a conflict in these checks, three sitemap corrections proposed and one availability investigation.” It is not “eight indexed pages.” The review has not established that.
After authorized fixes, regenerate the sitemap, retrieve the deployed file and recheck affected rows. Keep before-and-after values. For the temporary error, leave the task open until the page's intended state is confirmed; removing it solely to make a report look cleaner hides the question.
6. Verify Google's report without overstating the result
In Search Console, a successful sitemap status means Google fetched and processed the file without errors. Discovered pages are URLs parsed from it, not a count of confirmed indexed pages. The report covers sitemaps submitted through the report or API, so an absent row does not prove Google never discovered the file another way. Search Console's Sitemaps report
Use URL Inspection for important unresolved pages. Its indexed information and live test describe different states. The live test cannot predict Google's selected canonical; that field belongs to the indexed information. Record the inspection date and, where shown, Google's last crawl date so an earlier crawl is not mistaken for evidence about a change made today. Search Console's URL Inspection guide
Finish with the scope, files reviewed, specific corrections, remaining questions and next check. A useful sitemap audit leaves a reproducible account of what changed and why, without turning a clean file into a promise of rankings or traffic.