ROC AUC calculator validation report
Reproduce the tied-score ROC AUC example and inspect results from 534 numerical and input-validation checks.
How we checked the ROC AUC calculator
On 5 November 2026, the calculator passed 534 checks on synthetic score groups. For 522 valid inputs, its AUC agreed with an independent reference within an absolute tolerance of 10−12, and its positive and negative case totals matched exactly. It rejected all 12 invalid inputs.
Work through the calculator’s starting example
| Score | Positive cases | Negative cases |
|---|---|---|
| 1 | 0 | 2 |
| 2 | 1 | 1 |
| 3 | 1 | 0 |
There are six possible positive–negative pairs. In five pairs the positive case has the higher score. One pair ties at score 2 and contributes half a point. The AUC is (5 + 0.5) / 6 = 11/12, displayed as 0.91666667. Reversing the score direction gives 1/12.
Independent reference calculation
The calculator counts correctly ordered and tied pairs. The Python reference traces the ROC curve as the threshold falls. At each distinct score it adds all observations at that score together, then integrates the trapezoid between successive points. Python fractions keep the reference area exact throughout the calculation. The script invokes the same JavaScript module used by the calculator and compares the raw result before display rounding.
With the example above, the successive false-positive and true-positive rates are (0, 0), (0, 1/2), (1/3, 1), and (1, 1). The two segments with positive width have areas 1/4 and 2/3, which sum to 11/12. ROC AUC method reference
Checks and scope
- Six worked cases cover the published example, perfect and reversed rankings, all ties, unsorted rows with negative scores, and inversion of the default scores.
- 496 fixtures exhaust every eligible assignment of zero, one, or two cases and controls at each of three distinct scores, with both classes present and every score group occupied.
- Twenty seeded stress cases use 500 score groups each and large counts. The largest observed absolute AUC difference across all valid fixtures was 6.74 × 10−16 when rounded upward.
- Twelve invalid fixtures cover missing classes, repeated scores, empty groups, malformed rows, and numerical or input-size limits.
These checks establish arithmetic agreement for the tested implementation. They use grouped, fully observed binary outcomes with larger scores predicting the positive class. Assessing the calibration, clinical utility, or performance of a prediction model requires separate validation data and analyses.
Reproduce the checks
Download and extract the bundle, then run python3 verify_roc.py inside the extracted folder with Python 3.9+ and Node.js 18+ installed. It contains the calculator snapshot, independent reference code, expected values, and per-fixture results. The recorded run used Python 3.12.8 and Node.js 22.17.0.
Suggested citation: StatMedAnalysis. ROC AUC calculator validation report, version 1, 5 November 2026. https://www.statmedanalysis.com/validation/roc-tie-audit