Skip to content

Reduce memory overhead when resolving overlapping object masks - #636

Open
nyavana wants to merge 1 commit into
facebookresearch:mainfrom
nyavana:fix-635-boolean-overlap
Open

nyavana wants to merge 1 commit into
facebookresearch:mainfrom
nyavana:fix-635-boolean-overlap

Conversation

@nyavana

@nyavana nyavana commented Oct 12, 2026

Copy link
Copy Markdown

Fixes #635.

Why this change

When SAM3 resolves overlaps between object masks, its zero-background path converts yes/no masks into larger numeric tensors. The tracking callers then convert the result back to boolean. This uses extra memory for an output that only needs to say which pixels belong to each object.

This PR removes that avoidable memory overhead by keeping the masks boolean, while preserving the selected pixels used by existing tracking callers.

What changes

A small fast path in Sam3TrackerPredictor and Sam3VideoTrackingMultiplexDemo returns a boolean mask after the existing winner selection when the input is boolean and the background is 0 or 0.0. Floating masks and nonzero backgrounds retain their current behavior. Inputs are not modified.

Validation

The final regression file passed 260 cases on CPU and 260 on CUDA, with zero skips, using PyTorch 2.7.1+cu128 on an RTX 4070 Ti SUPER. Earlier Colab isolated-method checks also passed. These checks cover both helpers, confirm the same binary results as the existing path, and exercise fallback behavior.

In synthetic eager T4 benchmarks (PyTorch 2.11.0+cu130) with 16 masks at 1080p, peak helper allocation fell about 59% for background 0 and 36% for 0.0. Median helper time fell about 53-54% and 34-36%, respectively.

These results cover eager helpers; full-model/video performance, total device VRAM savings, and compiled behavior remain unmeasured. The test and tracker file pass formatting. The repository-wide formatting check reports the same pre-existing failures on pristine upstream main; upstream CI remains pending.

Run the regression test from the patched checkout in a provisioned SAM3 environment:

SAM3_TEST_DEVICES=cpu python -m pytest -q test/test_object_wise_non_overlapping.py
SAM3_TEST_DEVICES=cuda python -m pytest -q test/test_object_wise_non_overlapping.py

Specify the file explicitly because default discovery targets tests/, while this test is under test/. CUDA requires an available GPU; skipped cases do not establish GPU validation.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Meta Open Source bot. label Oct 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Meta Open Source bot.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Avoid int64 intermediate tensors in Boolean mask overlap postprocessing

1 participant