Prevent Leakage in Prediction Validation

Draw the information boundary first

Prediction validation asks how a complete modeling procedure performs on observations it has not already learned from. Write down the intended prediction moment and list which information would actually be available then. A variable recorded after the outcome, or a discharge code created after an admission outcome is known, can leak information even when the train–test split is technically correct.

The scikit-learn common-pitfalls documentation explains why preprocessing must learn from training data and then be applied to held-out observations. The same boundary applies to imputation, scaling, feature selection, and tuning. These steps are part of the model-development procedure rather than harmless preparation performed before validation.

Split at the appropriate unit

Decide whether the intended generalization concerns new observations, new people, new centers, or future time periods. Repeated visits from the same person can make a random row split misleading when the goal is prediction for new people. Retain a split manifest with identifiers, group membership, and assignment rules so overlapping units can be detected.

For a fictional example, imagine 100 people with three visits each. Assigning 240 rows to training and 60 rows to testing gives a nominal 80/20 row split, but does not guarantee that the test set contains unseen people. Assigning 80 people to training and 20 to testing instead yields the same 240/60 row counts while respecting person-level separation. Equal counts do not imply equivalent validation designs.

Fit every learned transformation inside the training step

Consider a second invented example with a training-variable mean of 10 in 80 records and a held-out mean of 30 in 20 records. The pooled mean is 14. Using 14 to impute training missing values has already allowed the held-out predictor distribution to influence the fitted procedure. The issue exists even though no held-out outcome was used.

Within cross-validation, refit preprocessing separately in each training fold. If hyperparameters or feature sets are selected, preserve an outer evaluation boundary when estimating the performance of that selection procedure. A pipeline helps maintain the order, but it cannot fix an incorrectly chosen grouping unit or a future-derived predictor.

Audit the final evaluation record

Save model version, split assignments, preprocessing parameters, tuning decisions, and the exact predictions scored. The prediction score worksheet can record the evaluation sample and probability convention before using the Brier tool for compatible aggregate inputs.

Keep unavailable outcomes and failed predictions visible. Repeatedly inspecting test performance and revising the model turns the test set into development information. If that happens, label the result accordingly and obtain a defensible new evaluation rather than presenting it as untouched external validation. A reproducible split is only one part of a reproducible information boundary.

Sources checked on 5 November 2026; all numerical examples are fictional and independently recomputed.