Colquhoun.AI
← All guides

Research Guides

Keep Rate-Limited Pages Out of Your Failure Count

· Colquhoun.shop editorial

A batch review of public pages can produce an apparently alarming result: several pages “failed.” Before turning that count into an SEO finding, separate a page defect from a request that the service declined because too many requests arrived.

This worksheet concerns authorized, read-only research and the reporting of incomplete batches. It is not a recipe for evading access controls, a load-testing guide or a claim about an automated Colquhoun.shop capability.

1. Record the response before interpreting the page

RFC 6585 defines HTTP 429 as a rate-limiting response. It may include a Retry-After header, and the standard does not prescribe how a server identifies a requester or counts requests. Therefore, one response does not reveal the site's exact quota or whether a limit is scoped to a page, account or broader service. RFC 6585, section 4: 429 Too Many Requests

Record the requested public URL, observation time and time zone, response status, relevant waiting instruction and the research job that made the request. Exclude authorization headers, cookies and private query strings from shared notes. Keep the observation “request rate-limited” distinct from a conclusion about the page's content or availability to other visitors.

2. Pause the affected work instead of multiplying it

Read the service's documented limits and any response instructions. Do not launch parallel retries, change identities or rotate addresses to get around the limit. If an authorized retry is appropriate, wait as instructed and use a bounded retry plan rather than an endless loop.

When the response provides no usable waiting instruction, record that uncertainty and stop or defer the affected work under a conservative plan. An absent header is not permission to immediately repeat the batch. If several jobs share a service, coordinate them so one retry does not collide with another job's requests.

3. Keep the denominator visible

Consider an invented review of 20 public pages: 14 were successfully read, four requests were rate-limited and two were not attempted after the pause. If three of the 14 readable pages lacked a required disclosure, report “3 of 14 successfully reviewed pages,” followed by the six incomplete observations. Do not call it “3 of 20 pages” without explaining that six were not assessed.

Likewise, the four rate-limited requests are not four confirmed content failures. Use separate fields for fetch outcome and editorial finding. An incomplete review can still be useful if readers can see exactly what it covers.

4. Resume without rewriting the history

If later access is permitted and a retry succeeds, append the new observation time and result. Keep the earlier rate-limit event in the working record. The content may have changed between observations, so do not present the completed batch as a single simultaneous snapshot when it was collected over different periods.

Deduplicate completed rows by your recorded page identity and research question. A second successful fetch should not silently add another page to the sample. If a redirected destination changes the identity, resolve that explicitly before updating totals.

5. Use a batch coverage ledger

For failures that are not specifically rate limits, use the broader unreadable-page research checklist. The central rule remains simple: an unsuccessful collection attempt is evidence about that attempt, not automatic evidence that the page lacks information or that the whole website is broken.

This guide describes the built-in Web Reader and Keyword Density behavior verified for this publication. Available data depends on the page response; these tools do not measure visits or search rankings.