Discrimination is not calibration. From-scratch, zero-dependency ROC / calibration / decision-curve evaluation that shows two predictors with byte-identical AUC but divergent, dangerous calibration — the statistical rigor most ML skips, every metric validated against a synthetic oracle.
· Python ≥ 3.9 · zero runtime dependencies · MIT
Read this first. A working proof-of-concept on synthetic data — not a clinical finding, not a validated risk score, not externally validated. The dataset comes from a known logistic risk equation precisely so every metric can be checked against ground-truth probabilities; that validation is the point. The "model" is deliberately secondary — a from-scratch gradient-descent logistic regression included for transparency, not performance. The edge is correct evaluation methodology. Uses "participant" throughout.
┌─ ROC AUC ─────────────► identical (0.7944 == 0.7944 == 0.7944)
one well-calibrated model ┤
+ a monotone distortion └─ Brier / ECE / calibration slope / intercept ─► diverge, then
snap back after recalibration
A model can rank participants perfectly (high AUC) yet output probabilities that are systematically wrong — which is exactly what makes a risk score unsafe at a decision threshold. This repo makes that undeniable.
metric well_calibrated miscalibrated recalibrated
-----------------------------------------------------------------
ROC AUC 0.7944 0.7944 0.7944
Brier 0.1718 0.1887 0.1719
ECE 0.0169 0.1117 0.0173
MCE 0.0503 0.2398 0.0500
cal slope 0.9892 0.4496 0.9879
cal intercept 0.0636 -0.1432 0.0641
Identical AUC (0.7944), yet 363 of 4000 participants (~9.1%) flip
treat/no-treat at threshold 0.5. AUC is not enough.
Reliability (well-calibrated): Reliability (miscalibrated):
obs | *. obs | .
obs | . obs | .
obs | .* obs | . *
obs | *.* obs | .
obs | *. obs | . * *
obs | *. obs | *. *
obs | *. obs | *. *
obs | *. obs | * *.
obs | *. obs | .
obs | *. obs | *.
obs |. obs |.
+--------------------- +---------------------
pred 0 1 pred 0 1
Well-calibrated points hug the diagonal; the miscalibrated model bows away while scoring the same AUC.
The miscalibrated predictor is constructed, not discovered: the well-calibrated model's own probabilities passed through a deliberate strictly-monotone map g(p) = sigmoid(2.2·logit(p) + 0.8).
- AUC is identical by mathematical necessity. AUC depends only on the ordering of scores.
gis a composition of strictly increasing functions on (0,1), so it preserves every concordant/discordant/tied pair exactly →AUC(g(p)) == AUC(p). Verified: difference is literally0.0(the test pins it at< 1e-12). - It is not a strawman.
a = 2.2 > 1produces textbook overconfidence (calibration slope < 1) — exactly what overfitting, shrinkage, and uncalibrated classifiers routinely produce in the wild. The consequence is real: ~9.1% of participants flip treat/no-treat at a 0.5 threshold between two models AUC calls equivalent. - Recalibration repairs it without touching AUC — because logistic recalibration is the same family of monotone map. Slope returns to ~0.99, ECE to ~0.017, AUC unchanged.
| Metric | Note |
|---|---|
roc_auc |
Mann-Whitney mid-rank rank-sum, exact half-credit for ties; nan (not 0.5) on a single class; validated against an O(n²) brute-force pairwise reference. |
brier_score + brier_decomposition |
Murphy reliability / resolution / uncertainty (binned; reconstructs Brier to ~1e-3, so the test tolerance is 0.02, not 1e-9). |
expected_/maximum_calibration_error |
Binning-dependent by nature — a known-gap test asserts the dependence rather than hiding it; always report n_bins + strategy. |
calibration_intercept_slope |
Slope = coefficient on logit(p) from a free 2-param IRLS fit. Intercept = calibration-in-the-large from an offset model with slope fixed at 1 (Cox / Van Calster). Conflating the two is the common error; a test asserts they differ. |
recalibrate |
Platt/logistic recalibration, fit on train, applied to test. |
net_benefit / decision_curve |
DCA net benefit = TP/n − (FP/n)·(pt/(1−pt)) vs treat-all / treat-none. |
bootstrap_ci |
Percentile CI, paired resampling, stdlib RNG. |
All calibration is evaluated out-of-sample on a held-out test split — in-sample, slope is exactly 1 and intercept exactly 0 by MLE stationarity (a tautology, pinned as a known-gap test).
Every metric is validated against analytic ground truth: AUC of perfect ranking = 1.0; AUC invariant under monotone transforms (diff = 0.0); Brier(const = prevalence) = prevalence·(1−prevalence) exactly; treat-all net benefit = 0 at pt = prevalence exactly; the oracle predictor yields ECE ≈ 0 and slope ≈ 1 out-of-sample; recalibration restores slope ≈ 1 without changing AUC. The from-scratch logistic regression recovers the data-generating coefficients ([0.84, −0.72, 0.53, 1.04] vs true [0.85, −0.70, 0.55, 0.90]).
Explicitly not claimed: any clinical utility or generalization, optimizer performance, nominal bootstrap coverage on small synthetic n, or that the synthetic prevalence/effects reflect any real population.
python selftest.py # zero-dependency proof (above)
python -m calibeval --json out.json # CLI: scorecard + machine-readable export
python -m pytest -q # metric, input-validation and oracle regressionsThe public evaluation functions reject empty or mismatched samples, non-binary outcomes and non-finite inputs with ValueError. Probability metrics require [0,1]; ranking metrics accept arbitrary finite scores. Calibration rejects unknown bin strategies and invalid bin counts. Bootstrap replicate counts and confidence levels are validated. Average precision evaluates all observations tied at a score together, so row order cannot change the result. An unconverged recalibration fit raises ValueError; only a positive fitted slope preserves ranking.
CI checks Python 3.9, 3.12 and 3.14, including the self-test, CLI JSON export, full suite and an isolated wheel import.
- DCA divergence is genuinely small here (~0.01 net-benefit gap; the miscalibrated model can even edge ahead at isolated thresholds). DCA is the "why it matters" consequence layer, not a second dramatic gap — and there is no pass/fail test on a model-vs-miscalibrated ordering. The undeniable divergence lives in ECE, Brier, and calibration slope/intercept; the honest clinical-consequence number is the decision-flip count.
- ECE / MCE / Murphy are binning-dependent; the bootstrap holds predictions fixed (evaluation-only, no bias correction, under-covers near AUC = 1 on small n). The optimizer is hand-rolled GD validated against the DGP, not a production solver.
- Synthetic outcome from a known equation — no real clinical signal.
MIT — see LICENSE. Built by Patrick Iben (p.iben@saeny.net). Companion projects: numguard · grounded-rag · guarded-llm-service · tool-grounded-agent.