Adjudicate Duplicate Observations Before Dropping Rows

Define duplication at the right level

Identical rows, repeated identifiers and repeated observations are different problems. Two identical laboratory results can come from separate specimens; two records with different values can be duplicate entries for the same specimen. Define the observation key before applying a duplicate-removal command.

The pandas duplicated reference documents selecting a subset of columns and marking all duplicates with keep=False. That is useful for building a review queue. Automatically retaining the first row is an ordering rule, not evidence that the first row is correct.

Inspect duplicate groups

Preserve source file, import batch, timestamps, specimen or encounter identifier, and any correction status. Group records by the proposed key and inspect differences across fields. Distinguish exact repeated export rows, amended records, technical replicates and separate encounters that the proposed key failed to distinguish.

Choose a resolution for each type. A later approved amendment may supersede an earlier record, whereas technical replicates may require a prespecified measurement-summary rule. Do not average conflicting entries just to make the group unique.

Follow a hypothetical reconciliation

Suppose 103 imported rows contain 100 unique participant-visit keys. Three keys each appear twice, creating three extra rows. Investigation finds one repeated export, one corrected measurement and one genuine second specimen collected at the same visit.

Removing all three extra rows by keeping the first record would mishandle at least the corrected measurement or second specimen unless the ordering happened to align with the intended rule. Instead, remove the repeated export copy, retain the approved correction with its history, and expand the key to include specimen identity where the analysis requires both specimens.

Verify the resulting analysis unit

After adjudication, rerun uniqueness checks on the revised key and reconcile row counts by resolution type. Keep excluded records in an audit file with reasons. A reduced row count should be explainable without relying on the software’s default behavior.

Check downstream joins and denominators because a duplicate can multiply rows elsewhere before anyone notices. If uncertainty remains about which record is correct, retain an unresolved status and examine its effect under justified alternatives. The final deliverable is a defensible set of observations with documented provenance, not simply a table that passes a uniqueness assertion after information has been discarded.

Source verification: 6 November 2026. Examples are hypothetical.