Timothy Chen1 · Ola Shorinwa1 · Joseph Bruno3 · Aiden Swann1 · Javier Yu1 · Weijia Zeng2 · Keiko Nagami1 · Philip Dames3 · Mac Schwager1
1Stanford University
2UC San Diego
3Temple University
Project Page | arXiv | Data
Splat-Nav is a real-time navigation pipeline designed to work with environment representations generated by Gaussian Splatting (GSplat), a powerful new 3D scene representation. Splat-Nav consists of two components: Splat-Plan, a safe planning module, and Splat-Loc, a robust pose estimation module.
Splat-Plan builds a safe-by-construction polytope corridor through the map based on mathematically rigorous collision constraints and then constructs a Bezier curve trajectory through this corridor. Splat-Loc provides a robust state estimation module, leveraging the point-cloud representation inherent in GSplat scenes for recursive real-time pose localization, given only RGB images. The most compute-intensive procedures in our navigation pipeline, such as the computation of the Bezier trajectories and the pose optimization problem run primarily on the CPU, freeing up GPU resources for GPU-intensive tasks, such as online training of Gaussian Splats. We demonstrate the safety and robustness of our pipeline in both simulation and hardware experiments, where we show online re-planning at greater than 5 Hz and pose estimation at about 25 Hz, an order of magnitude faster than Neural Radiance Field (NeRF)-based navigation methods, thereby enabling real-time navigation.
- Our ROS nodes for Splat-Plan and Splat-Loc can now be found in the
rosbranch of this repository. - Splat-Nav supports semantic goal querying. While in practice, Splat-Nav can use any semantic GSplats to query a goal, our results run on an in-house semantic GSplat model, which can be found in the
semanticsbranch of this repository.
This repository is built off of Nerfstudio. Please first follow the installation instructions there before installing any of the dependencies specifically for this repository. Once you have Nerfstudio installed in your Conda environment, install the following dependencies in that environment.
- Clarabel. This library is for solving the quadratic program.
- dijkstra3d. This library is for solving A* for path initialization.
- polytope. This library is for performing certain operations with polytopes.
- cvxopt. Generic convex solver.
- cvxpy. Generic convex solver.
- unfoldNd. Necessary to perform Maxpool3D operation over masked kernel for producing voxel grid.
- LightGlue. Feature matching for Splat-Loc.
- viser. Web-based 3D interactive visualizer. This already comes with Nerfstudio, however the latest version is needed to render Gaussian Splats.
Our datasets, trained models, and trajectories are hosted on a Google Drive. The scenes used in the paper are flightgate (flight), statues, stonehenge, adirondacks (which is also named old union). The training data and model is in the training folder, while the trajectories are in traj (simulated) and ros (hardware). You can drag and drop these folders into your working directory.
Here's an example:
Splat-Nav
├── data
│ └── flight
│ └── images
│ └── transforms.json
│
├──outputs
│ └── flight
│ └── splatfacto
│ └── 2024-09-12_172434
│ └── nerfstudio_models
| └── config.yml
| └── dataparser_transforms.json # This file contains the transform that transforms from "Data" frame to "Nerfstudio" frame (which is typically a unit box)
├── run.py
After the dependencies, data, and model is set up, run
python run_splatplan.py
to execute Splat-Plan on simulated scenes. The most important thing is to ensure that the path in GSplatLoader is pointing to the right model location. Many parameters can be tuned in this file. The trajectory data will be saved in the trajs folder.
Within run.py, there is also the option to run our Python implementation of the Safe Flight Corridor baseline (https://ieeexplore.ieee.org/document/7839930). The RRT* baseline can be executed through run_rrt.py.
Similarly, run
python run_splatloc.py
to execute Splat-Loc on specified trained models and datasets.
Data visualization is done through viser (https://github.com/nerfstudio-project/viser). It is a web-based 3D interactive visualizer that supports meshes and Gaussian Splatting. Viser also supports programmatic view rendering, which was used to generate many of the visuals in our videos. To visualize the trajectories in viser, run
python visualize.py
. The latest version of viser should be used, as that supports Gaussian Splatting rendering.
To use your own datasets, simply train a Nerfstudio splatfacto model and follow the folder structure as illustrated above. We also provide a simple way of producing sparse reconstructions of the scene by including a simple L1 sparsity loss on the opacity during Gaussian Splatting training (https://github.com/chengine/sparse_splat). These sparser models are also provided in the Google Drive. In run.py, there is a flag to use the sparser model.
If you found Splat-Nav useful, consider citing us! Thanks!
@misc{chen2024splatnav,
title={Splat-Nav: Safe Real-Time Robot Navigation in Gaussian Splatting Maps},
author={Timothy Chen and Ola Shorinwa and Joseph Bruno and Javier Yu and Weijia Zeng and Aiden Swann and Keiko Nagami and Philip Dames and Mac Schwager},
year={2024},
eprint={2403.02751},
archivePrefix={arXiv},
primaryClass={cs.RO},
url={https://arxiv.org/abs/2403.02751},
}