Binary prediction score audit worksheet
StatMedAnalysis | Version 2026-11-05

Instructions: save a working copy, fill blank fields, retain assumptions and
record an independent arithmetic check. Use fictional or aggregate values;
do not enter patient identifiers. This is a planning aid, not clinical advice.

1. Define validation
Model version and intended population/outcome/horizon: ____________
Validation sample and split date: ________________________________
Leakage checks, outcome completeness and exclusions: _____________
All evaluated models use the same sample? ________________________
Use hypothetical or non-identifying aggregate values only.

2. Build exact-probability groups
Each row must contain cases with EXACTLY the same prediction p.
Require 0 <= p <= 1, whole n > 0, and whole 0 <= e <= n.
Do not substitute a coarse bin mean for varied individual predictions:
that cannot recover the original individual Brier score.
p | event count e | total n | e(1-p)^2+(n-e)p^2 | n*p
__|_______________|_________|___________________|____
__|_______________|_________|___________________|____
Total N: ______ Sum events: ______ Sum loss: ______ Sum n*p: ______
Binary Brier = total loss/N (0–1 scale, smaller is better): ________
Observed proportion = events/N: ______ Mean prediction: __________
Overall observed-minus-predicted gap: ____________________________
Hypothetical rows .2,3,10 and .8,7,10 produce Brier .22 and overall
mean prediction=.5=observed proportion. Overall balance does not
establish calibration across probabilities; the row observed-minus-predicted gaps are +0.1 and -0.1.

3. Broader assessment
Calibration plot/bin definitions and row sample sizes: ___________
Discrimination, uncertainty and clinical utility analyses: ________
Prevalence and intended comparator: ______________________________
Brier combines calibration and discrimination; comparisons across
unrelated populations can mislead. This worksheet estimates no CI.

4. Report
Validation design, exact aggregation, units and limitations: ______
Prepared by/date/version: __________ Independent check: __________

Guide and primary-source links (verified 2026-11-05):
https://www.statmedanalysis.com/blog/brier-score-calibration
https://scikit-learn.org/stable/modules/generated/sklearn.metrics.brier_score_loss.html
https://www.statmedanalysis.com/tools/brier-calibration