Repeat the Model-Building Process When Estimating Optimism

Define the procedure being evaluated

Apparent performance evaluates a fitted model on the data used to build it and can be optimistic. Bootstrap optimism correction examines that tendency by repeatedly rebuilding the model in resampled development datasets. Write down the entire development procedure before starting the resampling.

The rms validation reference documents resampling validation and repeated model-fitting options. If the original procedure selected predictors or tuned settings, evaluating only a fixed final coefficient vector does not reproduce the same development process.

Keep each bootstrap comparison paired

For each bootstrap sample, run the planned development steps. Evaluate that bootstrap-fitted model in its own bootstrap sample and in the original development data, using the same performance definition. Their difference estimates optimism for that repetition when larger metric values indicate better performance.

Average these differences and subtract the average from the original model’s apparent performance. For loss measures where smaller is better, define the sign convention explicitly rather than copying the subtraction blindly. Save failed fits and separation warnings; silently excluding difficult resamples can distort the assessment.

Calculate a hypothetical correction

Suppose the original apparent C statistic is 0.82. Across a fictional set of bootstrap repetitions, the average difference between bootstrap-training and original-data evaluation is 0.04. The optimism-corrected estimate is 0.82 − 0.04 = 0.78.

This calculation does not mean that every future population will yield a C statistic of 0.78. It describes an internal-validation estimate under the sampling and development assumptions. Changes in patient mix, measurement, practice or outcome frequency require additional external evaluation.

Review stability and resampling design

Choose a sufficient replication count for the intended precision and inspect whether estimates stabilize as repetitions accumulate. Respect clustering or other dependence when defining resampling units. An ordinary participant bootstrap may be inappropriate when observations are repeated within people or centers.

Report the number of successful and failed repetitions, all repeated development steps, the metric and its sign convention. Keep the final model fitted to the full development data separate from the temporary bootstrap models used for evaluation. If the results suggest substantial overfitting, reconsider model complexity and sample adequacy rather than treating the corrected number as a repair to the model itself. The deliverable should include both a reproducible validation procedure and an honest statement of the population it can inform.

Source verification: 6 November 2026. Examples are hypothetical.