[vadims-git]

Summary

• Inference runs on Apple Silicon (MPS) and on CPU, not only CUDA. import sam3 works on macOS, and the device is picked automatically (CUDA, then MPS, then CPU) or set with SAM3_DEVICE.
• Fixes the SAM 3.1 start_session TypeError (Bug: start_session() raises TypeError with multiplex predictor (SAM 3.1) #544), which breaks sam3.1_video_predictor_example.ipynb on every platform.
• Declares psutil, which import sam3 needs (Missing dependencies #227, Missing modules after installation #252).

These are three separate commits, so each can be taken on its own.

What changed (commit 3)

• New sam3/device_utils.py: DEVICE and bf16_autocast() replace the ~90 hardcoded .cuda(), device="cuda" and torch.autocast(device_type="cuda") call sites. bf16 autocast stays on for every device, because the SAM 3.1 multiplex path casts features to bf16 explicitly.
• Guard the import-time torch.cuda.get_device_properties(0) call in sam3_multiplex_base.py.
• Import the triton EDT kernel lazily (fixes No module named 'triton' on Mac #179).
• Replace pkg_resources (dropped from setuptools>=81) with importlib.resources.
• addmm_act: off CUDA, compute the linear + activation unfused, since _addmm_activation has no MPS kernel. The fallback matches the reference kernel exactly.
• Disable FlashAttention 3 off CUDA; catch the AssertionError that torch.cuda.mem_get_info() raises on builds without CUDA; use torch.mps.empty_cache() on MPS.
• Skip decord on macOS in the notebooks extra (no wheels, and no notebook uses it). Document Apple Silicon/CPU setup in the README.

CUDA behavior should be unchanged: DEVICE is cuda there, and every rewritten call site resolves to the same device and autocast as before.

Testing

Apple M5 (16 GB), macOS 26, Python 3.12, torch 2.14:

• SAM 3.1 (build_sam3_multiplex_video_predictor, real facebook/sam3.1 weights): text and point prompts, propagation, close_session, on MP4, JPEG-folder and single-image inputs. All ops run natively on MPS (PYTORCH_ENABLE_MPS_FALLBACK=0). As a sanity check, "person" detections on two reference frames match hand counts (1 and 11) on both MPS and CPU. All 64 missing checkpoint keys are freqs_cis_* buffers computed at init, and there are no unexpected keys.
• SAM 3 image API (build_sam3_image_model + Sam3Processor): text prompt, box prompt and batched images, on MPS and CPU. This used random weights, because I don't have access to the facebook/sam3 checkpoint.
• Unit tests: 16 pass (11 existing, 5 new). The new ones cover start_session kwarg filtering (fails on main with the Bug: start_session() raises TypeError with multiplex predictor (SAM 3.1) #544 TypeError), the SAM3_DEVICE override, bf16 autocast, and the addmm_act fallback against the _addmm_activation reference for GELU and ReLU.
• Formatting: ufmt check sam3 scripts with the versions pinned in format.yml reports no new failures. main already has 58 files that fail it.
• Not tested on CUDA (no NVIDIA GPU available). A CUDA run would be appreciated.

On the M5, SAM 3.1 tracks 720p video at roughly 1.3 fps with a peak of ~7 GB; CPU is about 50x slower.

Not covered

• build_sam3_video_predictor (the SAM 3 multi-GPU video predictor) still requires CUDA.
• The torchcodec video loader is still CUDA-only; the default OpenCV loader works.
• Training code is unchanged.

Related

• Earlier Mac PRs: Support device agnostic image inference (mps, cpu etc) #173, Add Apple CPU support for inference #258, Add Apple Silicon (MPS) support #264, Add MPS support for image inference on macOS (Apple Silicon) #400, Fix macOS/CPU compatibility: lazy triton, replace pkg_resources & decord, fix hardcoded device="cuda" #482. This one uses the same ideas (for example the lazy triton import), extends them to the SAM 3.1 multiplex files added in SAM 3.1 Release #503, and applies cleanly to current main.
• Fix: apply inspect-based kwarg filtering to start_session() to fix multiplex predictor TypeError #543 fixes the same Bug: start_session() raises TypeError with multiplex predictor (SAM 3.1) #544 bug. Commit 1 can be dropped if that lands first.

Fixes #179, fixes #544.

🤖 Generated with Claude Code

[meta-cla]

Hi @vadims-git!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at cla@meta.com. Thanks!

[meta-cla]

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Meta Open Source project. Thanks!

[vadims-git]

Could a maintainer help get CI running on this PR? The SAM3/ufmt workflow is marked as failed, but no jobs ran. The CLA is signed. Could you check whether the workflow needs approval or a rerun?