Choose Calibration Bins Without Hiding Error

State the purpose of the display

A calibration plot compares predicted probabilities with observed outcome frequencies. It is a diagnostic display, not a certificate that a prediction model is safe or useful. Define the outcome horizon, validation population, and probability version before choosing bins. Keep this choice distinct from selecting a threshold for a clinical action.

The scikit-learn calibration-curve documentation describes uniform probability bins and quantile bins with approximately comparable sample counts. Uniform bins preserve a fixed probability scale; quantile bins may provide more balanced counts but can have very different widths. Neither makes sparse information disappear.

Preserve the information needed for each point

For every bin, retain the lower and upper boundaries, the boundary inclusion convention, number of observations, number of events, and mean predicted probability. Report empty bins rather than creating a point with zero observed risk. An empty interval contains no observations from which an event frequency can be estimated.

Use the mean predicted probability as the horizontal coordinate, rather than automatically placing the point at the midpoint of the bin. A bin spanning 0.20 to 0.40 could contain nearly all predictions close to 0.21. Plotting its midpoint at 0.30 would misrepresent the model's actual predictions.

See how pooling can conceal disagreement

In a fictional validation sample, 50 observations each have predicted probability 0.20 but 20 events occur, an observed frequency of 0.40. Another 50 each have predicted probability 0.80 but only 30 events occur, an observed frequency of 0.60. The two groups show opposite calibration discrepancies.

If both groups are pooled into one bin, the mean predicted probability is 0.50 and the observed frequency is also 0.50. The combined point sits on the ideal line even though neither group agrees with its prediction. This is a direct arithmetic illustration of information lost through aggregation, not a claim that one particular bin count is always best.

Keep scoring separate from plotting

The Brier and calibration tool can summarize rows with a common probability. For the two fictional rows, the exact binary Brier score is 0.28. Substituting the pooled mean probability 0.50 yields 0.25 for a different set of predictions. Therefore a coarse calibration table generally cannot reconstruct the original individual score.

Use the prediction score worksheet to preserve both plotting choices and score inputs. Inspect a small number of defensible alternative bin schemes as a sensitivity exercise, rather than searching until a plot looks favorable. Retain counts or a prediction distribution display alongside the curve.

Finish by stating where information is sparse and whether calibration was assessed on independent validation data. A smooth-looking curve or one pooled point does not replace uncertainty assessment, discrimination analysis, or evaluation of decisions made using the predictions.

Sources checked on 5 November 2026; all numerical examples are fictional and independently recomputed.