Liquidation data: audit what a reported event actually measures · 4 / 5
Remove a proven liquidation replay without deleting genuine observations
A reconnect or retried processing job can make a liquidation chart jump twice for one stored observation. Removing every row with an identical timestamp, side, size and price is not a complete solution: genuinely distinct events can share those fields. The useful question is which duplication the available provenance actually proves. Local replay evidence and provider event identity solve different problems.
Athenum8 minUpdated:
Separate three kinds of identity
A transport or storage identifier can establish that your own system processed the same saved envelope twice. A provider event identifier can establish that two deliveries refer to the same underlying event, if its documented scope and stability support that conclusion. A fingerprint of ordinary event values establishes only that those values match. A fingerprint is not automatically an event identifier.
The public liquidation schemas referenced here do not expose a general account identifier or a stable unique execution identifier for every observation. Do not invent one from the subscription request ID or the timestamp. Record the confidence of any deduplication rule, preserve raw inputs and distinguish removed local replays from unresolved equal-valued provider observations.
Preserve a correction trail
For a research total, retain the original processing count, the proven replay quantity, the corrected local total and the unresolved cases. If a heuristic changes the total, report the rule and its effect. A clean-looking single number is not more reliable merely because uncertainty was hidden. Late updates, cumulative order snapshots and revised source records also require explicit semantics; they cannot all be handled as identical duplicates.
Twenty processed units become fifteen after one proven local replay
In this fictional archive, a partition and offset together identify one immutable stored envelope. Envelope P0:41 contains two rows of sizes 2 and 3; a worker processes that same stored envelope twice. Envelope P0:42 contains one size-5 row. Envelope P0:43 contains another size-5 row with identical public event fields, but no provider event identifier is available. Both later envelopes are independently stored observations. All sizes use the same base unit and are incremental for this example.
Naive processing adds 5 + 5 + 5 + 5 = 20 units. The repeated read of P0:41 is proven by the local archive identity, so remove its second five-unit contribution: the local-once total is 15. The equality of P0:42 and P0:43 does not establish whether they describe one event delivered twice or two distinct events. If they are the same underlying event, the underlying quantity is 10; if distinct, it is 15. Those alternatives describe this closed fictional ledger, not a bound on the complete exchange market.
| Processing pass | Stored envelope | Quantity | Evidence-based treatment |
|---|---|---|---|
| 1 | P0:41 | 2 + 3 = 5 | Retain the first local processing |
| 2 | P0:41 | 2 + 3 = 5 | Remove this proven replay |
| 3 | P0:42 | 5 | Retain; provider identity unresolved |
| 4 | P0:43 | 5 | Retain; equal fields do not prove identity |
Open full-size diagram- Naive processing total: 20 base units
- Proven replay contribution: 5 base units
- Each stored envelope counted once: 15 base units
- If equal rows share one event: 10 base units
A local offset proves local identity only
Two different offsets prove two stored records, not necessarily two unique exchange events. Conversely, a content hash can merge two distinct stored events whose public fields happen to match. Preserve enough context to explain both limitations. If independent evidence later resolves the two later records, revise the statistic with a dated correction rather than silently changing historical research inputs.
Before acting
- Distinguish archive identity from provider event identity.
- Remove only replays established by the selected evidence.
- Preserve raw fields and the applied correction rule.
- Keep equal-valued observations unresolved when identity is absent.
- Report the original total, correction and remaining uncertainty.
Check your understanding
A reliable later provider reconciliation confirms that P0:42 and P0:43 describe one event. What is the final quantity for this fictional ledger, and how much was the original 20-unit total overstated?
Show the explained answer
The final quantity is 10 units: five from P0:41 and five from the one later event. The original 20-unit processing total exceeded it by 10 units, or 100% of the corrected 10-unit total. This resolves the fictional ledger only; it does not prove complete exchange coverage.