Mixed Reality UAV Control for Infrastructure Inspection
Full YouTube video here: https://youtu.be/jKusI9VYvzU
🎉 Accepted to ICRA 2026: This project paper, titled “3DGS-Holo-Inspector: A Mixed Reality UAV Controller with 3D Gaussian Splatting Localization for Infrastructure Inspection” has been accepted for presentation at the IEEE International Conference on Robotics and Automation (ICRA 2026) in Vienna, Austria 🇦🇹. (Link to the paper will be released in June 2026)
This repository contains the Mixed Reality (MR) application for 3DGS-Holo-Inspector — a system that enables holographic goal definition and autonomous UAV navigation using Mixed Reality. The MR-App allows operators to intuitively define, preview, and adjust UAV goals in 3D space via a Microsoft HoloLens 2 interface.
3DGS-Holo-Inspector as a whole combines:
- MR App (this repository) — defines and previews holographic goals and displays real-time telemetry.
- [Localization Server](Git Repo here) — performs dense, markerless alignment between MR headset and UAV using 3D Gaussian Splatting (3DGS).
- Drone Control Module — transforms MR goals into UAV commands and executes autonomous navigation.
- Holographic goal placement for UAV navigation
- Hand-gesture interaction for goal definition and adjustment
- Pre-flight visualization of UAV poses in real 3D space
- Real-time overlay of UAV telemetry and camera feed in MR
- Designed for use in infrastructure inspection tasks
Hardware
- Microsoft HoloLens 2
- PC (runs the Localization Server and the Drone Control Module)
Software
- Unity 2020.3.24 or compatible LTS version
- xr_ros
- ROS-TCP-Connector
- OpenXR-Unity-MixedReality-Samples
- A network link between HoloLens 2 and the ground control workstation
-
Clone this repository
git clone https://github.com/cviss-lab/3dgs-holoinspector.git cd 3dgs-holoinspector -
Open in Unity
- Open Unity Hub
- Add project folder
- Open with Unity 2020.3.24
-
Import MRTK
- Import the Mixed Reality Toolkit packages
- Configure the scene for HoloLens 2
-
Configure Networking
- Ensure HL2 can communicate with your ROS network
- Use ROS-TCP-Connector to bridge Unity and ROS
- mention the IP address where the
xr_rospackage will be running with the ROS master
-
Build & Deploy
- Build the MR App for Universal Windows Platform (UWP)
- Deploy to the HoloLens 2 via Visual Studio
-
Setup the xr_ros package
- Clone the
xr_rospackage's branchgs-localizationand confgure according to its README. git clone https://github.com/cviss-lab/xr_ros -b gs-localization
- Clone the
-
Setup the localization server
- Clone the localization server package and configure according to its README.
-
Setup the drone control module
- Clone the
sentinel-controlpackage and configure according to its README. git clone https://github.com/cviss-lab/sentinel-control.git
- Clone the
- HoloLens 2 runs this MR App.
- Operator places holographic goals using hand gestures.
- The MR App sends goal poses to the ground station via ROS, which is received by the
xr_rospackage - The HL2 and UAV use the
xr_rospackage to communicate with the localization server to establish the spatial alignment between the HL2, UAV, and the dataset map. - The Drone Control Module transforms goals using Localization Server alignments.
- UAV autonomously executes the goal, and real-time info is streamed back to the MR view.
This work is part of the 3DGS-Holo-Inspector system developed at the University of Waterloo.
Supported by Rogers Communications Inc., Canada and the Natural Sciences and Engineering Research Council of Canada (NSERC).
If you have questions or want to collaborate, contact:
- Syed Muhammad Raza Rizvi — smr4rizv@uwaterloo.ca
- Huaiyuan Weng — huaiyuan.weng@uwaterloo.ca
- Chul Min Yeum — cmyeum@uwaterloo.ca
Have a look at the Computer Vision for Smart Structure Lab at the University of Waterloo!
Contributions, issues, and feature requests are welcome!