Research Guides
Preserve Large Research Identifiers Before Parsing and Joining
A research export can be valid JSON and still lose the distinction between two record identifiers when a downstream program reads it. If those identifiers connect citations, review records or source rows, the result can be a false match that looks like a successful import. Test identity preservation before joining or deduplicating the data.
This is a data-handoff worksheet, not a report of a defect in Colquhoun.shop. The example uses invented identifiers and a specific JavaScript parsing path; other parsers can behave differently.
1. Decide whether the field is a measurement or an identity
A count of visits is a quantity. A provider's record identifier is a reference, even when it contains only digits. Adding two identifiers has no useful research meaning. Keep that distinction in the data contract rather than allowing a spreadsheet or parser to infer it from appearance.
RFC 8259's number guidance permits implementations to limit numeric range and precision. It identifies integers from negative (2 raised to 53 minus 1) through positive (2 raised to 53 minus 1) as an interoperability range where implementations agree exactly. A syntactically valid JSON number outside that range is not a guarantee of exact preservation through every consumer.
2. Reproduce a collision before using real records
Use a synthetic fixture containing two number-valued fields: {"a":9007199254740992,"b":9007199254740993}. In a JavaScript test using ordinary JSON.parse and Number values, both parsed fields become 9007199254740992. The source text contains two distinct integers; that parsing path no longer preserves the difference.
Now test a string-valued fixture: {"a":"9007199254740992","b":"9007199254740993"}. Keep the fields as strings throughout the test. They remain distinct without requiring arithmetic on the identifiers. This is an example contract choice, not permission to change an external API's expected types without coordinating its producer and consumers.
The failure can occur before a join starts. Turning an already rounded value into a string afterward only preserves the wrong value. Recover the original source rather than trying to guess the missing digit.
3. Trace the identity through every handoff
- Source: preserve a safe synthetic fixture and its exact identifier text.
- Parse: record the runtime, parser and resulting field type.
- Transform: inspect number conversions, sorting and normalization.
- Join: compare distinct-key counts before matching records.
- Export: reopen the output with the intended consumer and compare exact identifiers.
Run both fixtures through the real pipeline used for the handoff. A unit test of the initial parser will miss a later conversion in an export helper. Include an identifier with leading zeros if the source contract permits them; that is a separate reason to avoid casually treating identifiers as quantities.
4. Repair the contract, not the observed result
For a system you control, agree on a lossless identifier representation with every consumer. If an external source requires numeric identifiers, use an appropriate lossless parsing path and test it end to end. Do not silently rewrite a provider's payload and then claim compatibility because one sample imported.
Keep arithmetic measurements separately typed and validated. Making every field a string can conceal other errors and does not explain how totals should be calculated.
5. Publish a narrow verification statement
A useful acceptance note names the fixture, the pipeline and the result: “The two synthetic identifiers remain distinct after parsing, joining and reopening the export in the tested consumer.” Retain the raw fixture with the test so another reviewer can reproduce it.
This establishes identity preservation for the tested path. It does not prove that the source records are authentic, that every join is logically correct or that all large-number formats are supported. Those are separate checks, which should not disappear behind a successful JSON parse.