Skip to content

Optimize Metropolis2D IoU evaluation - #313

Open
raus7n wants to merge 1 commit into
NVIDIA:mainfrom
raus7n:optimize-metropolis2d-iou
Open

Optimize Metropolis2D IoU evaluation#313
raus7n wants to merge 1 commit into
NVIDIA:mainfrom
raus7n:optimize-metropolis2d-iou

Conversation

@raus7n

@raus7n raus7n commented Aug 11, 2026

Copy link
Copy Markdown

Summary

This PR optimizes the Metropolis2D 2D detection evaluation by precomputing the prediction-to-ground-truth IoU matrix once per image and reusing it across all IoU thresholds.

Motivation

The evaluator uses 10 IoU thresholds from 0.50 to 0.95. Previously, the same prediction/ground-truth IoU pairs were recomputed independently for each threshold.

For N predictions, M ground-truth boxes, and T IoU thresholds, this results in approximately:

O(T × N × M)

IoU calculations.

Changes

  • Precompute the N × M IoU matrix once for each image.
  • Reuse the precomputed IoUs during matching across all IoU thresholds.
  • Preserve the existing confidence ordering and greedy matching logic.
  • Preserve the existing per-threshold ground-truth matching behavior.
  • Leave the AP/mAP calculation unchanged.

This reduces the number of IoU calculations from approximately:

O(T × N × M)

to:

O(N × M)

where T = 10.

Validation

  • python -m py_compile evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/dataset/metropolis2d/detection_2d_dataset.py
  • git diff --check

Testing

The existing evaluation logic and matching semantics are unchanged. The optimized implementation was syntax-checked successfully.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant