Audit Binary Prediction Scores

Define the outcome and prediction time

Before calculating a prediction score, specify which event is coded one, the prediction horizon, and when each predicted probability was available. A probability for an outcome within one year cannot be scored against an outcome measured at an inconsistent follow-up time without an appropriate handling strategy.

The scikit-learn Brier documentation defines the commonly used binary score as the mean squared difference between probability p and outcome y. This convention ranges from zero to one, with lower values indicating smaller squared prediction error. Explicitly state the convention because multiclass and unscaled formulations can use a different range.

Check whether aggregate inputs are sufficient

The Brier and calibration tool accepts rows containing a common predicted probability, event count, and total count. For a row with n observations, e events, and identical probability p for everyone, its squared-error sum is e×(1−p)²+(n−e)×p². Add these sums and divide by the combined number of observations.

This calculation is exact for those common-probability rows. A bin's average predicted probability is generally insufficient to recover the original individual-level Brier score when probabilities vary inside the bin. Do not substitute a mean and call the result exact unless that shared-probability assumption actually holds.

Recompute a fictional score

Suppose 50 people each receive probability 0.20 and ten experience the outcome. Another 50 each receive probability 0.80 and 40 experience it. The first row contributes 10×0.8²+40×0.2² = 8. The second contributes 40×0.2²+10×0.8² = 8. The total Brier score is 16/100 = 0.16.

Both rows have observed frequencies equal to their stated probabilities. The overall event frequency is 0.50, and a constant prediction of 0.50 for every person has score 0.25 in this fictional dataset. That comparison provides context; it is not a universal benchmark that transfers to populations with different event frequencies.

Separate calibration from overall score

Calibration asks whether predicted probabilities agree with observed frequencies. The Brier score also reflects how predictions separate outcomes and the variability of the outcome. A lower Brier score alone does not prove improved calibration.

Use the prediction score worksheet to save the event definition, validation sample, probability convention, and aggregate-input justification. Examine calibration across the probability range, not only the overall mean prediction. Sparse rows require particular care because their observed frequencies are unstable.

Evaluate predictions on data not used to fit or tune the model, and compare candidate models on the same eligible observations. Report missing predictions and unavailable outcomes. A numerical score can be reproducible while still answering the wrong evaluation question if selection, timing, or leakage has changed the validation population.

Sources checked on 5 November 2026; all numerical examples are fictional and independently recomputed.