Treat Recalibration as a Model Update That Needs Evaluation

Diagnose the problem before updating

A model can rank people reasonably well while systematically overpredicting or underpredicting absolute risk. Recalibration may address some such discrepancies, but it should follow a documented validation assessment. Save the original predictions and their performance before fitting any update.

The scikit-learn calibration guide explains probability calibration and the need to separate fitting and calibration data appropriately. Updating probabilities using the same outcomes used to judge the update can produce an overly favorable performance estimate.

Choose the smallest justified update

For an existing binary-risk model, an intercept update changes the overall level on the log-odds scale while holding the original slope contribution fixed. A logistic intercept-and-slope update changes both level and spread. More flexible calibration methods require enough data and can introduce their own overfitting.

Specify the population, outcome horizon and update method in advance. Investigate whether discrepancies instead arise from a coding error, different outcome definition, unavailable predictor or incorrect unit conversion. Recalibration should not hide a broken implementation of the original model.

Work through a hypothetical intercept change

Suppose a model predicts risk 0.20, corresponding to odds 0.20/0.80 = 0.25. An illustrative intercept update adds ln2 to every predicted log odds, doubling those odds to 0.50. The updated risk is 0.50/(1 + 0.50), approximately 0.333.

A prediction of 0.80 has odds 4 and becomes odds 8, or risk approximately 0.889. The update is not an addition of the same number of percentage points to every probability. Its effect depends on the original risk, which is why storing the exact transformation matters.

Evaluate the updated prediction rule

Use a suitable held-out, resampling or later evaluation strategy that accounts for estimating the update. Report calibration, discrimination and decision-relevant performance for both original and updated versions. An intercept-only monotone update can leave ranking unchanged while materially changing threshold-based decisions.

Version the updated model, including the original model identifier, calibration dataset, fitted parameters and intended population. Record how future data drift will be monitored and what would trigger another review. If the available update sample is small or unrepresentative, describe the uncertainty rather than presenting a smooth calibration curve as a guarantee of transportability. Recalibration is a new estimation step with its own evidence requirements, not a cosmetic correction to a published score.

Source verification: 6 November 2026. Examples are hypothetical.