Skip to content

Add automatic confidence-threshold calibration to both training pipelines - #61

Open
nilshof01 wants to merge 1 commit into
hotosm:developfrom
nilshof01:feat/threshold-calibration
Open

Add automatic confidence-threshold calibration to both training pipelines#61
nilshof01 wants to merge 1 commit into
hotosm:developfrom
nilshof01:feat/threshold-calibration

Conversation

@nilshof01

Copy link
Copy Markdown

…ines

The optimal mask threshold is not a constant: measured across 3 regions x 2 models x 5 folds (size 32) it ranged 0.10-0.90, and the shipped operating points (UNet fixed 0.5, DINOv3 catalog 0.4371) cost +1.1 to +13.9 test F1 vs a deterministic 17-point val-split sweep, which landed within 0.7 F1 of the test-oracle in every measured cell.

unet_segmentation:

  • new calibrate_threshold step: val-split F1 sweep, rate-matching on the train sampler when val has no positive pixels (threshold at which the predicted positive-pixel fraction matches the labeled fraction)
  • predict() now thresholds the most likely foreground class directly; the previous argmax+top-prob rule made binary thresholds below 0.5 no-ops (identical output at the 0.5 default)

dinov3s_buildings:

  • new calibrate_threshold step (same sweep on the spatial val chips via cache_val_forwards; rate-matching on train chips when val < 2 chips)
  • tune_postprocess defaults are seeded with the calibrated threshold, so the skipped-search path (val < 8 chips or trials disabled - i.e. every small-dataset run) serves the calibrated value instead of the catalog constant. When Optuna runs it still tunes the threshold jointly in [0.3, 0.8]; fixing it there would require a dinov3_hot change.

Both STAC items declare training.calibrate_threshold (default true).

…ines

The optimal mask threshold is not a constant: measured across 3 regions
x 2 models x 5 folds (size 32) it ranged 0.10-0.90, and the shipped
operating points (UNet fixed 0.5, DINOv3 catalog 0.4371) cost +1.1 to
+13.9 test F1 vs a deterministic 17-point val-split sweep, which landed
within 0.7 F1 of the test-oracle in every measured cell.

unet_segmentation:
- new calibrate_threshold step: val-split F1 sweep, rate-matching on the
  train sampler when val has no positive pixels (threshold at which the
  predicted positive-pixel fraction matches the labeled fraction)
- predict() now thresholds the most likely foreground class directly;
  the previous argmax+top-prob rule made binary thresholds below 0.5
  no-ops (identical output at the 0.5 default)

dinov3s_buildings:
- new calibrate_threshold step (same sweep on the spatial val chips via
  cache_val_forwards; rate-matching on train chips when val < 2 chips)
- tune_postprocess defaults are seeded with the calibrated threshold, so
  the skipped-search path (val < 8 chips or trials disabled - i.e. every
  small-dataset run) serves the calibrated value instead of the catalog
  constant. When Optuna runs it still tunes the threshold jointly in
  [0.3, 0.8]; fixing it there would require a dinov3_hot change.

Both STAC items declare training.calibrate_threshold (default true).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@nilshof01 nilshof01 changed the title Add automatic confidence-threshold calibration to both training pipel… Add automatic confidence-threshold calibration to both training pipelines Jul 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant