Validate a clinical prediction model
Check whether model predictions hold up in the intended patient population. Start with the validation sample, then examine ranking, probability accuracy and performance at a decision threshold.
Worked decision
Suppose a model assigns 20% risk to 100 people with complete follow-up and 10 experience the outcome. The observed risk in this group is 10%. Investigate the discrepancy and its uncertainty before adopting a 20% treatment threshold, even when the model ranks patients well.
Record your decision
Would you use these predictions at the proposed threshold in this population? Record where the model was tested, whether predicted risks matched observed outcomes, and how the rule compared with treating everyone or no one.
Work through the evidence
- Check how the validation sample was separated
- Calculate ROC AUC from grouped scores
- Assess probability accuracy and calibration
- Compare net benefit at the proposed threshold
- Download the prediction-score worksheet (TXT)
- Read the scikit-learn guide to calibration curves