Research Guides
Check the Response Format Before Relaxing a Research Parser
A research import fails with a JSON parsing error, and the first proposed fix is to make the parser more forgiving. But the saved response begins with an HTML sign-in page. The parser may be correctly rejecting content that was never the promised dataset.
This guide provides a response-format triage record for website research. It does not claim that Colquhoun.AI exposes arbitrary API requests or automatically diagnoses import failures.
1. Separate the format declaration from the received content
MDN's Content-Type reference explains that a response's Content-Type identifies the media type of the returned data. It distinguishes this from Content-Encoding, which concerns decoding the content to its original form. A charset parameter can identify the character encoding.
Record the declared type, but do not stop there. The declaration and the received representation can disagree. A JSON-looking URL, a successful HTTP status or a familiar filename does not establish that the expected research data arrived.
2. Trace a fictional redirected export
Imagine a fictional export address expected to return a JSON array of page observations. A request follows a redirect to a login screen. The final response is HTTP 200 with text/html, and the importer reports an unexpected opening character.
The reviewer preserves the requested address, redirect destination, final status and declared type. A small, safely inspected excerpt confirms that the body is a sign-in page rather than observations. The useful finding is an access or routing mismatch, not evidence that the JSON parser needs to accept HTML.
The next step is to use the documented authorized access flow or ask the service owner to correct the route. It is not to bypass authentication, submit invented credentials or scrape private material from another session.
3. Classify before changing the importer
- Unexpected destination: the response came from a different page or access flow.
- Unexpected declared format: the final Content-Type differs from the documented contract.
- Format mismatch: the declaration says one thing while the safely inspected content indicates another.
- Invalid expected format: the intended representation arrived, but its syntax is malformed.
- Valid syntax, wrong structure: parsing succeeds, but required research fields or types are absent.
These categories lead to different repairs. Do not collapse them into a single failed-import count if the report needs to explain what actually prevented the research task.
4. Inspect without creating another exposure
Use an appropriate non-executing inspection method for unfamiliar responses. Do not open unknown active content merely to discover its format. Keep credentials, session cookies and private records out of shared debugging notes.
Retain only the evidence needed to reproduce the mismatch within the authorized workflow. Where a response contains customer data, prefer a sanitized fixture for parser tests. A fixture should preserve the relevant structural defect without reproducing personal details.
5. Copy this response-format triage card
- Expected contract: documented format and required structure.
- Request: address, method and safe access context.
- Redirects: final destination and any relevant intermediate change.
- Response: status, Content-Type and relevant encoding metadata.
- Observed body: safe evidence of the actual representation.
- Failure category: routing, declaration, syntax or structure.
- Repair owner: service configuration, access flow or importer.
- Retest: expected data received and validated without weakening checks.
Keep the original failed case as a sanitized regression test when feasible. A repaired import should still reject an HTML login page presented where a research dataset is required.
6. Report the task-level outcome
A successful retest shows that the intended representation arrived through authorized access, parsed correctly and matched the expected structure. It does not establish that every research value inside it is accurate.
Separating transport, format and content validation makes the incident easier to resolve without turning a clear failure into silently corrupted evidence.