Keep Model Selection Inside the Inner Cross-Validation Loop

Identify every data-dependent choice

Nested cross-validation separates selecting a prediction procedure from evaluating that selection process. List hyperparameter tuning, feature selection, transformations, imputation and threshold selection before deciding how to split the data. Any step that learns from outcomes or predictor distributions must be placed appropriately within the workflow.

The scikit-learn nested-validation example contrasts nested and non-nested evaluation. Using the same validation results both to select a model and to report its performance can produce an optimistic estimate of generalization.

Give the loops different jobs

The outer loop reserves data for evaluation. Within each outer training set, the inner loop compares candidate configurations using only that training set. After a configuration is selected, fit the corresponding pipeline on the full outer training set and predict the untouched outer test set.

Keep preprocessing inside the fitted pipeline so the inner and outer splits do not share learned transformations improperly. Use group-aware or time-aware splitting when the data structure requires it. Random row splits are not appropriate merely because they produce convenient balanced folds.

Trace a hypothetical five-by-four design

Suppose 100 fictional independent observations are divided into five equal outer folds. For one outer iteration, 80 observations form the training set and 20 are held out. Four-fold inner validation splits the 80 into 60 for inner fitting and 20 for inner checking at each iteration.

After tuning, the selected configuration is refitted on all 80 outer-training observations and evaluated on the untouched 20. Repeat this process for the five outer folds. The outer outcomes must not influence which configuration is selected within their corresponding iteration.

Summarize the process honestly

Combine or summarize outer predictions using metrics appropriate to the task, and report the splitting scheme and variability. Fold scores are not generally independent observations, so a naive standard error obtained by dividing their SD by the square root of the fold count can be misleading.

For a final deployable model, apply the prespecified selection procedure using the available development data and fit the chosen pipeline. Its coefficients need not match any one outer model. Preserve the distinction between evaluating a model-building process and evaluating a single already-frozen model. Record random seeds, grouping rules, candidate settings and failed fits so another analyst can reproduce the choices rather than just the final performance number.

Source verification: 6 November 2026. Examples are hypothetical.