VIGS-SLAM: Visual Inertial Gaussian Splatting SLAM

Zihan Zhu · Wei Zhang · Moyang Li · Norbert Haala · Marc Pollefeys · Daniel Barath

ECCV 2026

Paper | Video | Real-Time Demo | Project Page | Dataset

Given a sequence of RGB frames and IMU readings, our VIGS-SLAM robustly tracks the camera trajectory while reconstructing a high-fidelity 3D Gaussian map.

Table of Contents

1. Getting Started
2. Run Demo
3. iPhone Capture & Run
4. Dataset Preparation
5. Batched Run & Tracking Evaluation
6. Citation
7. Acknowledgements
8. Contact

Getting Started

NOTE: A Docker setup is available — see docker/README.md for prebuilt-image and online GPU direct template load instructions.

1. Clone the repo with submodules

git clone --recursive https://github.com/cvg/VIGS-SLAM

2. Create a new Conda environment and then activate it. Here we provide the environment for RTX 5090 (should also work on other GPUs). Please note that we use the PyTorch version 2.8 compiled by CUDA 12.8 in the environment_5090.yaml file.

conda env create -f environment_5090.yaml
conda activate vigs-slam-5090
pip install thirdparty/simple-knn
pip install thirdparty/diff-gaussian-rasterization
python -m pip install --no-binary=:all: -U sophuspy
pip install --no-binary=:all: --no-build-isolation --no-cache-dir -v torch-scatter

3. Compile the CUDA kernel extensions (takes about 10 minutes). Please note that this process assumes you have CUDA 12.8 installed. To check the installed CUDA version, you can run nvcc --version in the terminal.

python setup.py install

4. Download the pretrained weights of Omnidata models for generating depth and normal priors

wget https://zenodo.org/records/10447888/files/omnidata_dpt_normal_v2.ckpt -P pretrained_models
wget https://zenodo.org/records/10447888/files/omnidata_dpt_depth_v2.ckpt -P pretrained_models

Note: The DroidNet checkpoint pretrained_models/droid.pth is bundled with this repository. It is taken from DROID-SLAM (BSD-3-Clause) and redistributed here under the same license.

5. (Optional) Build IMU C++ module

The IMU C++ module (vigs/imu_cpp/) significantly accelerates IMU factor computation compared to the pure Python implementation. If not built, the system falls back to pure Python automatically.

Prerequisites & Build Instructions

Install the following system packages if not already present:

# C++ compiler and CMake build system
sudo apt update
sudo apt install build-essential cmake

# Eigen3 linear algebra library (header-only)
sudo apt install libeigen3-dev

# OpenMP runtime (usually bundled with GCC; install explicitly if using Clang)
sudo apt install libomp-dev

Note: pybind11 is provided by the conda environment set up in step 2, and Sophus (header-only) is vendored as a git submodule under thirdparty/Sophus — neither needs a separate/global installation. Make sure the environment is activated before running cmake, and that the submodule is present (git submodule update --init thirdparty/Sophus).

cd vigs/imu_cpp
mkdir -p build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release
make -j$(nproc)

The compiled module imu_integrator_cpp*.so will be placed in vigs/imu_cpp/build/. VIGS-SLAM auto-detects and loads it at runtime — no config change needed.

Note (build flags): CMakeLists.txt compiles with -DSOPHUS_DISABLE_ENSURES. This is required, not optional — the IMU code constructs Sophus::SO3d directly from rotation blocks of pose matrices (see imu_integrator.cpp), which carry small numerical drift. Sophus's default SO3(Matrix3d) constructor runs a runtime orthogonality check that calls std::abort() on such matrices; the module explicitly relies on that check being disabled. If you build the module without this flag, the run will abort with Sophus ensure failed ... R is not orthogonal during IMU initialization. (-DSOPHUS_USE_BASIC_LOGGING is also set to avoid Sophus's optional fmt dependency.)

6. (Optional) TensorRT Acceleration

TensorRT accelerates neural network inference by optimizing and compiling models into GPU-specific execution engines, delivering significantly lower latency than standard PyTorch.

Prerequisites & Build Instructions

Since TensorRT engines are device-specific, they must be built on each machine. Install TensorRT following the official NVIDIA installation guide. Make sure the version is compatible with your CUDA and PyTorch versions. We use TensorRT 10.13.0.35 with CUDA 12.8 on the RTX 5090.

Once installed, first export the pretrained Omnidata models to ONNX, then convert them to TensorRT .engine files:

Step 1 — Export to ONNX:

python scripts/export_omnidata.py
python scripts/export_droidnet.py

Step 2 — Simplify ONNX graphs (requires pip install onnxsim):

cd pretrained_models
python -m onnxsim omnidata_depth_512.onnx  omnidata_depth_512_simplified.onnx  --skip-fuse-bn
python -m onnxsim omnidata_normal_512.onnx omnidata_normal_512_simplified.onnx --skip-fuse-bn

Step 3 — Build TensorRT FP16 engines:

• --memPoolSize=workspace:N — scratch memory (MB) TensorRT can use during engine build and inference. Increase if build fails; decrease if GPU memory is limited.
• --minShapes / --optShapes / --maxShapes — define the range of input shapes for dynamic engines. TensorRT optimizes for optShapes and supports any shape within [min, max]. Set these based on the actual input shapes your sequences will produce — too wide a range reduces optimization quality.

cd pretrained_models
# Omnidata depth & normal
trtexec --onnx=omnidata_depth_512_simplified.onnx \
        --saveEngine=omnidata_depth_512_simplified_fp16.engine \
        --memPoolSize=workspace:24576 --fp16

trtexec --onnx=omnidata_normal_512_simplified.onnx \
        --saveEngine=omnidata_normal_512_simplified_fp16.engine \
        --memPoolSize=workspace:24576 --fp16

# DroidNet feature encoder (dynamic resolution)
trtexec --onnx=droidnet_fnet.onnx \
        --saveEngine=droidnet_fnet_fp16.engine \
        --memPoolSize=workspace:24576 --fp16 \
        --minShapes=input:1x1x3x328x328 \
        --optShapes=input:1x1x3x368x584 \
        --maxShapes=input:1x1x3x656x656

# DroidNet update module — frontend window
trtexec --onnx=update_module_partial.onnx \
        --saveEngine=update_module_partial_fp16.engine \
        --memPoolSize=workspace:24576 --fp16 \
        --minShapes=net:1x1x128x41x41,inp:1x1x128x41x41,corr:1x1x196x41x41,flow:1x1x4x41x41 \
        --optShapes=net:1x24x128x43x77,inp:1x24x128x43x77,corr:1x24x196x43x77,flow:1x24x4x43x77 \
        --maxShapes=net:1x60x128x82x82,inp:1x60x128x82x82,corr:1x60x196x82x82,flow:1x60x4x82x82

# DroidNet update module — backend / PGBA / final BA (larger window)
trtexec --onnx=update_module_partial.onnx \
        --saveEngine=update_module_partial_pgba_fp16.engine \
        --memPoolSize=workspace:24576 --fp16 \
        --minShapes=net:1x1x128x41x41,inp:1x1x128x41x41,corr:1x1x196x41x41,flow:1x1x4x41x41 \
        --optShapes=net:1x85x128x43x77,inp:1x85x128x43x77,corr:1x85x196x43x77,flow:1x85x4x43x77 \
        --maxShapes=net:1x120x128x82x82,inp:1x120x128x82x82,corr:1x120x196x82x82,flow:1x120x4x82x82

The generated .engine files will be placed in pretrained_models/. VIGS-SLAM auto-detects them at runtime and falls back to PyTorch if they are not found.

Trouble Shooting

1. CUDA extensions built for the wrong GPU architecture. The build should automatically detect your GPU's compute capability and specify the matching gencode flags. If it does not (e.g. building on a host with no GPU visible, or hitting a no kernel image is available for execution on the device error at runtime), you may need to manually add your architecture to the CUDA extension setup.py files. For example, for an RTX 5090 (sm_120) add -gencode=arch=compute_120,code=sm_120 to the nvcc compile args in:

   • thirdparty/diff-gaussian-rasterization/setup.py
   • thirdparty/simple-knn/setup.py

   Alternatively, export TORCH_CUDA_ARCH_LIST="12.0" (or "8.9;12.0" for both RTX 4090 and 5090) before building.

Run Demo

The demo uses the first 1000 frames of the RPNG AR Table (table_01) sequence. Results are saved to outputs/demo/, including estimated camera poses, the Gaussian map, and renderings. Optional flags: --gsvis to watch the Gaussian map build in real time, --droidvis to inspect intermediate depth and point cloud estimates.

First, download the demo data:

bash scripts/prep_demo.sh

Then run:

python demo.py \
--imagedir data/demo/rgb \
--calib calib/rpngar.txt \
--config config/rpng.yaml \
--imufile data/demo/imu.txt \
--undistort \
--IMU_poseinit_after 20 \
--output outputs/demo \
[--gsmapping]    # Optional: enable Gaussian mapping
[--pure_online]  # Optional: online-only mode — skips final BA, trajectory filling, and Gaussian map refinement
[--input_vis]    # Optional: show input image in a cv2 window
[--gsvis]        # Optional: enable Gaussian map display (OpenGL window)
[--droidvis]     # Optional: enable point cloud display (OpenGL window)
[--rerunvis]     # Optional: live visualization in the Rerun web viewer
[--rerun_record] # Optional: save a .rrd file for later replay with Rerun

Note: --rerunvis and --rerun_record are mutually exclusive — if both are given, live streaming takes priority and no .rrd file is saved.

Live visualization with Rerun (--rerunvis): once the run starts, open the printed URL in your browser (e.g. http://127.0.0.1:9876?url=...). If running on a remote server, forward both ports via SSH first:

ssh -L 9876:127.0.0.1:9876 -L 9877:127.0.0.1:9877 <your_server>

Replay a saved Rerun recording (--rerun_record):

rerun outputs/demo/rerun_stream.rrd

iPhone Capture & Run

1. Install the iOS recorder app

A custom iOS recorder app is required to stream RGB frames and IMU data to the capture server. The source code and a short installation instruction video are available at the PhoneStreamer repository — see its README for installation instructions.

2. Capture a session

Make sure your iPhone and your machine are on the same local network. Find your machine's local IP:

ifconfig

Enter that IP in the iOS app. The port is preset to 9999 on both sides.

Then start the capture server on your machine:

python scripts/stream_data.py

In the iOS app, tap Start. The first ~1 second is used for auto-focus and locking the intrinsics, after which the app begins streaming RGB frames and IMU data. Tap Stop when done.

This saves a timestamped session folder under data/iphone/, containing:

• images/ — JPEG frames named by timestamp (nanoseconds)
• imu.csv — synchronized IMU data
• calib.txt — camera intrinsics (fx fy cx cy)
• camera.json — full camera metadata

3. Run Sequence

Open run_iphone_mono.py and add your session folder name to SEQS:

SEQS = [
    "20260309_170728",   # replace with your session folder name
]

Also review the other options at the top of the file (DATA_ROOT, output_folder, gsmapping, etc.), then run:

python run_iphone_mono.py

Results (trajectory, Gaussian map) will be saved under outputs/output_iphone/.

Alternative: Live SLAM Streaming

Instead of the two-step capture-then-run flow above, demo_stream.py runs VIGS-SLAM in real time as the iPhone streams. No data is saved to disk first — SLAM starts immediately after the first frame arrives.

Make sure your iPhone and machine are on the same local network, then run:

python demo_stream.py \
--config config/rpng.yaml \
--output outputs/stream \
--IMU_poseinit_after 25 \
[--gsmapping]         # Optional: enable Gaussian mapping
[--gsvis]             # Optional: Gaussian map display (OpenGL window)
[--droidvis]          # Optional: point cloud display (OpenGL window)
[--rerunvis]          # Optional: live visualization in the Rerun web viewer

Open the iOS app, enter your machine's local IP and port 9999, then tap Start. SLAM begins automatically once intrinsics and an initial IMU backlog are received. Tap Stop to end the session.

When the iPhone disconnects, the system runs a final bundle adjustment and saves:

• traj_kf_beforeBA.txt — keyframe trajectory before final BA
• traj_kf_afterBA.txt — keyframe trajectory after final BA
• 3dgs_before_final.ply / 3dgs_final.ply — Gaussian map before and after final BA
• intrinsics.npy — scaled camera intrinsics used during the session

Note: Unlike the offline pipeline, demo_stream.py receives camera intrinsics directly from the iPhone — no --calib file is needed.

---

Dataset Preparation

Note: By downloading any of the following datasets, you agree to the terms and conditions set by the respective dataset providers. Please review each dataset's license before use.

EuRoC Dataset

Download the processed EuRoC Dataset using the script below

bash scripts/prep_euroc.sh

RPNG AR Table Dataset

Download the processed RPNG AR Table Dataset using the script below

bash scripts/prep_rpng.sh

UTMM Dataset

Download the processed UTMM dataset using the script below

bash scripts/prep_utmm.sh

FAST-LIVO2 Dataset

Download the processed FAST-LIVO2 dataset using the script below

bash scripts/prep_livo2.sh

VIGS-SLAM Dataset

Download our self-captured VIGS-SLAM dataset using the script below

bash scripts/prep_vigs.sh

Batched Run & Tracking Evaluation

Each script runs all sequences in batch. Set the dataset paths, output directories, and whether to enable Gaussian Mapping at the top of each file before running.

Due to massive refactoring, cleaning, and optimization, results may not be exactly the same as in the paper, but are generally better. Differences across environments and GPU models have also been observed.

EuRoC Dataset

Note: Place all sequences directly under a single base folder (e.g. data/euroc/MH_01_easy/, data/euroc/V1_01_easy/). Do not nest them in sub-categories such as Machine Hall/ or Vicon Room 1, Vicon Room 2/ — the evaluation script expects a flat layout.

Note: The EuRoC ground-truth trajectories used for evaluation are provided in euroc_groundtruth/. These are taken from DROID-SLAM and are expressed in the image (camera) coordinate frame (rather than the original EuRoC body/IMU frame), so they align directly with the estimated camera trajectories.

python eval_euroc_mono.py

RPNG AR Table Dataset

python eval_rpng_mono.py

UTMM Dataset

python eval_utmm_mono.py

FAST-LIVO2 Dataset

python eval_fastlivo_mono.py

VIGS-SLAM Dataset

python eval_vigs_mono.py

Citation

If you find our code, paper or dataset useful, please cite

@inproceedings{ZHU2026VIGS-SLAM,
      title={VIGS-SLAM: Visual Inertial Gaussian Splatting SLAM},
      author={Zhu, Zihan and Zhang, Wei and Li, Moyang and Haala, Norbert and Pollefeys, Marc and Barath, Daniel},
      booktitle={European Conference on Computer Vision (ECCV)},
      year={2026}
}

Acknowledgements

We adapted some codes from some awesome repositories including HI-SLAM2, DROID-SLAM, 3D Gaussian Splatting, MonoGS, and Omnidata/MiDaS. Thanks for making codes public available. We thank Ke Wu for help with VINGS-Mono and Codey Sun for help with MM3DGS-SLAM. We are grateful to Anusha Krishnan and Shaohui Liu for valuable discussions, and to Boyang Sun and Zuria Bauer for providing the capture site for the demo.

License

This project is released under the Apache 2.0 license. Some of the adapted components are covered by their own, and in some cases more restrictive, licenses, so different parts of this repository may be under different licenses. Please review and comply with the license of each original project before using the corresponding parts of this repository.

Contact

Contact Zihan Zhu for questions, comments and reporting bugs.