Deformable Instance MetalSplatter

This project is forked from an amazing work MetalSplatter. It provides support for running deformable splats as well as instance segmented splats.

Render deformable 3D Gaussian Splats using Metal on Apple platforms. Tested on :

• Iphone 15, Iphone 17 Air
• Ipad Pro (M1)
• Macbook Pro (M5)

Installation

Please follow the steps in original README to setup the project in xcode.

TODOS

• add optional speedup via static vs dynamic splat masking (export_static_mask.py ready, Swift integration pending)
• Update BibTex after 3DV proceedings are published
• ...

Usage

By selecting a folder in the startup page, the app loads the weights.bin, clusters.bin and point_cloud.ply inside the directory. You can download example scenes from here trained with TRASE.

There is a scroll bar for adjusting the time but you can also let it play by deactivating the manual time setting. You can toggle the additional dropdown to unlock additional options such as the TRASE based instance segmentation "Show Clusters" and depth based visualisation "Depth" (based on camera viewpoint). Clicking on clusters will isolate them in the visualisation, showing the whole scene again can be done via "Show all" button.

The gestures for X/Y Panning, Orbit, Zoom in/out are also implemented. As some scenes (such as sear-stake) may have a flipped coordinate system we add some buttons to change the coordinate axis in the dropdown.

Using your own Scenes

Step 1: Create env

conda create -n trase_model_converter python=3.10 -y
conda activate trase_model_converter
pip install torch "numpy<2" torchinfo

Step 2: For dynamic splat scenes

Output should be the base path of your input folder, where point_cloud.ply is stored.

Export the deform.pth via :

python export_deform_weights.py --model <path-to-deform.pth> --output <path to deform output.bin>

Step 3 TRASE clustering: Optionally for clusters (store in same path as point_cloud.ply just like for 1) :

Train your scene with TRASE and run the export script :

python export_clusters_bin.py --model <clusters.pt> --output <path to clusters output.bin>

Step 4 MobileCLIP querying: Optionally for cluster scenes (perform step 3 first)

Download the Mobileclip coreml models from our google drive link and put it in DeformableMetalSplatter/ (whole path is then DeformableMetalSplatter/coreml_models/)

You can then move to a certain view and run the Encode Clusters CLIP button to encode the features, after which the search bar can be used to query clusters with text. topk setting can be used to decide which amount of clusters can be queried.

Demo

Dynamic Splats + Instances on iPhone 17 Air

Iphone_sear_stake.mp4

Dynamic Splats + Instances on iPad Pro (M1)

ipad_split_cookie.mp4

Dynamic Splats + Instances on iPhone 15

as_novel_view_iPhone15.mp4

MobileCLIP Encoding and Instance Querying on iPhone 17 Air

ScreenRecording_02-14-2026.1-03-50.PM_1.mov

MobileCLIP Encoding and Instance Querying on Macbook Pro M5

sear.steak.mov

painting.mov

Acknowledgments

This project is a fork of MetalSplatter created by Sean Cier.

Original code is licensed under the MIT License (Copyright © 2023 Sean Cier).

Modifications and new features are licensed under MIT License (Copyright © 2026 Jim Li).

References

If you find this useful for your own work, particularly the 3D segmentation please consider taking a look at TRASE's code, paper, and website. Give us a star if you find it interesting :)

@article{li2024trase,
    title={TRASE: Tracking-free 4D Segmentation and Editing},
    author={Li, Yun-Jin and Gladkova, Mariia and Xia, Yan and Cremers, Daniel},
    journal={arXiv preprint arXiv:2411.19290},
    year={2024}
}

Contributors