Keep Raw Research Exports Separate from Analysis Corrections

Define what counts as raw

A raw export is the file as received from a specified source at a specified time. It may already contain errors, derived fields or system-level transformations. Calling it raw does not certify its scientific correctness; it identifies the starting point for your own analysis pipeline.

Keep that starting point unchanged. Store cleaned and analysis-ready files in separate locations with explicit names and provenance. The Python file-opening documentation distinguishes read, write and append modes; opening the source in a writing mode can truncate or alter the evidence you intended to preserve.

Separate correction authority from implementation

When a source value is wrong, record the issue and obtain the appropriate correction evidence. Prefer a corrected source-system export when available. If an analysis-layer correction is necessary, maintain a structured patch containing the record key, field, original value, replacement, reason, evidence and date.

Apply patches through reproducible code with an assertion that the original value matches the expected value. That assertion prevents an old correction from being applied again after the source system has already fixed the record.

Reconcile a hypothetical update

Suppose export A contains 1,000 rows and one confirmed date error. The analysis pipeline applies one documented correction and produces a cleaned file with the same 1,000 rows. Later export B contains 1,020 rows and already includes the corrected date.

A patch that blindly overwrites the date would conceal whether the source had changed. A patch that requires the old erroneous value should stop or report that the correction is no longer applicable. The reviewer can then retire the patch for export B while preserving its role in analyses based on export A.

Verify recoverability

Record file hashes, source timestamps and import settings. Hashes can detect byte changes but do not explain them; keep the transformation log and authorized source context as well. Restrict write access to the raw layer according to the project’s controls rather than relying only on a folder name such as do-not-edit.

Test whether a clean working directory can recreate the analysis-ready dataset from the preserved export, configuration and patches. Compare row counts, keys and important derived values. The outcome should be a recoverable process in which every correction is visible, not a pristine-looking final spreadsheet whose history depends on undocumented manual edits.

Source verification: 6 November 2026. Examples are hypothetical.