The Data-Cleaning Log Is Part of the Scientific Record

A final analysis dataset can look clean enough to conceal how many scientific decisions were required to produce it. Dates were corrected, duplicate records removed, units converted, impossible values queried, free text recoded, and participants excluded. If those actions exist only in email, spreadsheet colors, or the memory of one analyst, the study cannot fully explain how raw observations became reported results.

A data-cleaning log preserves that transformation history. It is not a list of every keystroke. It is a versioned record of decisions and actions that could affect data meaning, participant inclusion, outcome derivation, or analysis.

01Separate source correction from analytical transformation

When an authorized review confirms that a source value was entered incorrectly, the corrected record should retain the original value, revised value, reason, date, person making the change, and supporting query or evidence. An audit trail should make the history visible.

Analytical transformations are different. Converting units, calculating age, defining baseline, mapping codes, or deriving a composite outcome may leave the source unchanged. These operations should be implemented in versioned code where possible, with the specification and test evidence linked from the log.

Do not overwrite raw data to make analysis easier. Preserve an immutable or access-controlled source extract and produce derived datasets reproducibly. This allows the team to rerun cleaning when new data arrive and to determine which rule created any value.

02Log the decisions with scientific consequences

Material entries include:

• correction of a value or date;
• duplicate-identification and retention rule;
• unit conversion or harmonization;
• code mapping and dictionary version;
• range or consistency query and resolution;
• handling of partial dates;
• derivation or algorithm change;
• merge, linkage, or record-matching rule;
• exclusion of a participant, observation, or outlier;
• missing-value recode;
• manual adjudication; and
• change to a cleaning rule after data review.

Each entry should identify the issue, affected variables and records, original and resulting state, rule or rationale, evidence, code or query reference, data version, author, reviewer, approval if required, and downstream outputs that may need regeneration.

03Use rules before case-by-case judgment

Cleaning is safest when decisions are specified as reusable rules. Instead of “remove implausible weights,” define the permitted range, how age or context affects it, whether the value is queried, and what happens if it cannot be resolved. An unusual observation should not be deleted merely because it makes a model fit worse.

MHRA data-integrity guidance states that data exclusion requires valid scientific justification, that excluded data should be retained, and that processing records should permit reconstruction. The exact regulatory obligations vary by study, but the research principle is general: unfavorable or inconvenient observations do not disappear without an inspectable reason.

04Make the log executable where possible

Code can apply deterministic cleaning rules, generate discrepancy reports, and create a machine-readable audit table. Manual decisions still occur, particularly for source queries or clinical adjudication. Store them in a controlled decision file that code reads, rather than editing the analysis dataset directly.

For each release, record input hashes or immutable version identifiers, code commit, dictionary version, manual decision-file version, output identifier, execution date, and person running the pipeline. A second analyst should be able to recreate the same dataset from the same inputs.

05Review patterns, not only individual queries

The log is also a quality signal. Repeated unit errors at one site may indicate training or interface problems. A surge in partial dates may follow a form revision. Many manual recodes of one variable may reveal an inadequate dictionary. Summarize cleaning events by type, site, time, and variable, and feed the findings back into collection.

06Preserve the record through reporting

Database lock does not make the cleaning history irrelevant. Link final tables and figures to the exact analysis dataset. If a post-lock correction occurs, reopen under controlled authorization, document the reason, rerun affected outputs, and assess whether the manuscript or conclusions change.

Begin with a simple log containing twelve fields: entry ID, date, issue, record scope, variable, original state, action, rationale, evidence, code/version, decision owner, and reviewer. If a reported number cannot be traced through those records to a defined source, data cleaning is not yet part of the scientific record—it is an undocumented intervention in it.

This article is educational and intended for research purposes. It does not provide individual medical advice, diagnosis, or treatment. No patient data were used.