Skip to content

About

Discrimination is not calibration: from-scratch ROC, calibration, and decision-curve evaluation validated against a synthetic oracle.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

3 Commits

Folders and files

Repository files navigation

calibrated-clinical-ml

Discrimination is not calibration. From-scratch, zero-dependency ROC / calibration / decision-curve evaluation that shows two predictors with byte-identical AUC but divergent, dangerous calibration — the statistical rigor most ML skips, every metric validated against a synthetic oracle.

ci  ·  Python ≥ 3.9 · zero runtime dependencies · MIT

Read this first. A working proof-of-concept on synthetic data — not a clinical finding, not a validated risk score, not externally validated. The dataset comes from a known logistic risk equation precisely so every metric can be checked against ground-truth probabilities; that validation is the point. The "model" is deliberately secondary — a from-scratch gradient-descent logistic regression included for transparency, not performance. The edge is correct evaluation methodology. Uses "participant" throughout.


The idea

                            ┌─ ROC AUC ─────────────► identical (0.7944 == 0.7944 == 0.7944)
  one well-calibrated model ┤
  + a monotone distortion   └─ Brier / ECE / calibration slope / intercept ─► diverge, then
                                                                               snap back after recalibration

A model can rank participants perfectly (high AUC) yet output probabilities that are systematically wrong — which is exactly what makes a risk score unsafe at a decision threshold. This repo makes that undeniable.

Real self-test output (python selftest.py)

  metric            well_calibrated   miscalibrated    recalibrated
  -----------------------------------------------------------------
  ROC AUC                    0.7944          0.7944          0.7944
  Brier                      0.1718          0.1887          0.1719
  ECE                        0.0169          0.1117          0.0173
  MCE                        0.0503          0.2398          0.0500
  cal slope                  0.9892          0.4496          0.9879
  cal intercept              0.0636         -0.1432          0.0641

  Identical AUC (0.7944), yet 363 of 4000 participants (~9.1%) flip
  treat/no-treat at threshold 0.5. AUC is not enough.
  Reliability (well-calibrated):        Reliability (miscalibrated):
   obs |                   *.            obs |                    .
   obs |                  .              obs |                  .
   obs |                .*               obs |                .  *
   obs |             *.*                 obs |              .
   obs |           *.                    obs |            .  * *
   obs |         *.                      obs |         *.  *
   obs |       *.                        obs |       *.  *
   obs |     *.                          obs |   * *.
   obs |   *.                            obs |    .
   obs | *.                              obs | *.
   obs |.                                obs |.
       +---------------------                +---------------------
        pred 0              1                 pred 0              1

Well-calibrated points hug the diagonal; the miscalibrated model bows away while scoring the same AUC.


Why two models with the same AUC?

The miscalibrated predictor is constructed, not discovered: the well-calibrated model's own probabilities passed through a deliberate strictly-monotone map g(p) = sigmoid(2.2·logit(p) + 0.8).

  • AUC is identical by mathematical necessity. AUC depends only on the ordering of scores. g is a composition of strictly increasing functions on (0,1), so it preserves every concordant/discordant/tied pair exactly → AUC(g(p)) == AUC(p). Verified: difference is literally 0.0 (the test pins it at < 1e-12).
  • It is not a strawman. a = 2.2 > 1 produces textbook overconfidence (calibration slope < 1) — exactly what overfitting, shrinkage, and uncalibrated classifiers routinely produce in the wild. The consequence is real: ~9.1% of participants flip treat/no-treat at a 0.5 threshold between two models AUC calls equivalent.
  • Recalibration repairs it without touching AUC — because logistic recalibration is the same family of monotone map. Slope returns to ~0.99, ECE to ~0.017, AUC unchanged.

What's inside (every metric implemented by hand)

Metric Note
roc_auc Mann-Whitney mid-rank rank-sum, exact half-credit for ties; nan (not 0.5) on a single class; validated against an O(n²) brute-force pairwise reference.
brier_score + brier_decomposition Murphy reliability / resolution / uncertainty (binned; reconstructs Brier to ~1e-3, so the test tolerance is 0.02, not 1e-9).
expected_/maximum_calibration_error Binning-dependent by nature — a known-gap test asserts the dependence rather than hiding it; always report n_bins + strategy.
calibration_intercept_slope Slope = coefficient on logit(p) from a free 2-param IRLS fit. Intercept = calibration-in-the-large from an offset model with slope fixed at 1 (Cox / Van Calster). Conflating the two is the common error; a test asserts they differ.
recalibrate Platt/logistic recalibration, fit on train, applied to test.
net_benefit / decision_curve DCA net benefit = TP/n − (FP/n)·(pt/(1−pt)) vs treat-all / treat-none.
bootstrap_ci Percentile CI, paired resampling, stdlib RNG.

All calibration is evaluated out-of-sample on a held-out test split — in-sample, slope is exactly 1 and intercept exactly 0 by MLE stationarity (a tautology, pinned as a known-gap test).


Why a statistician built this

Every metric is validated against analytic ground truth: AUC of perfect ranking = 1.0; AUC invariant under monotone transforms (diff = 0.0); Brier(const = prevalence) = prevalence·(1−prevalence) exactly; treat-all net benefit = 0 at pt = prevalence exactly; the oracle predictor yields ECE ≈ 0 and slope ≈ 1 out-of-sample; recalibration restores slope ≈ 1 without changing AUC. The from-scratch logistic regression recovers the data-generating coefficients ([0.84, −0.72, 0.53, 1.04] vs true [0.85, −0.70, 0.55, 0.90]).

Explicitly not claimed: any clinical utility or generalization, optimizer performance, nominal bootstrap coverage on small synthetic n, or that the synthetic prevalence/effects reflect any real population.


Quickstart

python selftest.py                  # zero-dependency proof (above)
python -m calibeval --json out.json # CLI: scorecard + machine-readable export
python -m pytest -q                 # metric, input-validation and oracle regressions

The public evaluation functions reject empty or mismatched samples, non-binary outcomes and non-finite inputs with ValueError. Probability metrics require [0,1]; ranking metrics accept arbitrary finite scores. Calibration rejects unknown bin strategies and invalid bin counts. Bootstrap replicate counts and confidence levels are validated. Average precision evaluates all observations tied at a score together, so row order cannot change the result. An unconverged recalibration fit raises ValueError; only a positive fitted slope preserves ranking.

CI checks Python 3.9, 3.12 and 3.14, including the self-test, CLI JSON export, full suite and an isolated wheel import.

What this does NOT do (honest limitations)

  • DCA divergence is genuinely small here (~0.01 net-benefit gap; the miscalibrated model can even edge ahead at isolated thresholds). DCA is the "why it matters" consequence layer, not a second dramatic gap — and there is no pass/fail test on a model-vs-miscalibrated ordering. The undeniable divergence lives in ECE, Brier, and calibration slope/intercept; the honest clinical-consequence number is the decision-flip count.
  • ECE / MCE / Murphy are binning-dependent; the bootstrap holds predictions fixed (evaluation-only, no bias correction, under-covers near AUC = 1 on small n). The optimizer is hand-rolled GD validated against the DGP, not a production solver.
  • Synthetic outcome from a known equation — no real clinical signal.

License

MIT — see LICENSE. Built by Patrick Iben (p.iben@saeny.net). Companion projects: numguard · grounded-rag · guarded-llm-service · tool-grounded-agent.

About

Discrimination is not calibration: from-scratch ROC, calibration, and decision-curve evaluation validated against a synthetic oracle.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages