Colquhoun.AI
← All guides

Research Guides

Check Robots Rules on the Host That Serves the Page

· Colquhoun.shop editorial

A robots.txt review can look correct while inspecting the wrong host. A main website, a documentation subdomain and a separate download host may each serve different crawl rules. Start with the exact page URL rather than assuming the main domain's file governs every related address.

This is a manual audit method for authorized website review. It does not change crawl settings, test private endpoints or describe an automated Colquhoun.shop feature.

1. Map the locations that actually serve content

Create one row per relevant combination of protocol, hostname and port. Include only locations established by your authorized inventory or public links, not speculative subdomains. Next to each row, record a representative article, asset or download URL and why it belongs in the review.

In an invented example, the main site is https://example.com, documentation is on https://docs.example.com and reports are on https://files.example.com. Treat them as separate rows. A shared brand name is not evidence of shared crawl-rule scope.

2. Retrieve the corresponding top-level file

Google documents that robots.txt rules apply to the host, protocol and port where the file is hosted. The file belongs at the top level; a file under a content subdirectory does not govern that directory as a separate robots.txt policy. Its documentation also explains that explicit default ports are equivalent to their usual implicit form. Google: robots.txt specification and scope

For each inventory row, record the robots.txt URL actually requested, response status, any redirects, retrieved rules and check time. A screenshot of a file in an editor is not evidence that the same content is served publicly at that address.

3. Test a named crawler and a concrete path

Write the crawler being evaluated and the exact target path. Review the applicable rule group using that crawler's documented interpretation. Do not combine unrelated groups into one imagined rule or assume every crawler follows every directive in the same way.

Keep an expected result beside the observed result for each test case. For a documentation move, include a retained public guide, a moved guide and any path that should remain restricted from crawling. Test rules without repeatedly fetching disallowed content.

4. Follow the content location without losing the starting point

If an article redirects from the main site to the documentation host, keep both URLs in the record and review the relevant hosts rather than replacing the original address silently. Redirect behavior and crawl-rule behavior are separate observations.

If a robots.txt request fails or returns unexpected content, record that result instead of inventing the missing rules. Use the crawler's documented handling to interpret the response, and keep any uncertainty visible. Do not treat a failed read as proof that the intended file was deployed.

5. Save an origin-to-rule matrix

This matrix establishes which crawl rules were checked where. It does not establish indexing, privacy or authorization to access protected content. A complete result should name the tested locations rather than declaring “the site's robots file is fine” after reviewing only the homepage's host.

This guide describes the built-in Web Reader and Keyword Density behavior verified for this publication. Available data depends on the page response; these tools do not measure visits or search rankings.