Neuro3D-Agent: AI Copilot for 3D Medical Imaging & Segmentation Analysis

An advanced, privacy-compliant AI assistant designed for radiologists to orchestrate 3D volumetric medical imaging workflows. The system integrates a Transformer-based Vision pipeline (Swin UNETR) for 3D semantic segmentation with an Agentic Orchestrator (LangChain + Local Llama 3 via Ollama) to dynamically parse clinical queries, process high-dimensional spatial data, and generate structured diagnostic reports.

📖 New to the domain or the stack? See CONCEPTS.md for a full glossary of the medical imaging and engineering concepts used in this project.

---

🚀 Key Architectural Pillars

3D Spatial Transformers

Built on top of the MONAI (Medical Open Network for AI) framework, utilizing preprocessing spatial orientations (RAS) and volumetric calculations on native 3D structural data.

Agentic Orchestration

Powered by LangChain, executing a structured tool-calling loop that treats deep learning inference models as dynamic runtime tools.

On-Premise Privacy

Uses a completely local LLM stack (Ollama + Llama 3), helping satisfy HIPAA-oriented deployment requirements by ensuring sensitive health information (PHI) remains within local infrastructure.

Modern Performance Backend

Asynchronous file-processing engine powered by FastAPI, paired with a responsive Streamlit user interface.

---

🛠️ Tech Stack & Dependencies

Category	Technologies
Package Manager	uv
Deep Learning & Medical Imaging	PyTorch, MONAI, Nibabel
Agent Framework	LangChain, Ollama
Backend	FastAPI, Uvicorn
Frontend	Streamlit

---

📁 Repository Structure

neuro3d-agent/
│
├── backend/
│   ├── main.py          # Asynchronous FastAPI application
│   └── agent_utils.py   # LangChain orchestration logic & tool bindings
│
├── models/
│   └── inference_3d.py  # MONAI pipeline & spatial matrix metrics extraction
│
├── frontend/
│   └── app.py           # Streamlit conversational web dashboard
│
├── uploaded_scans/      # Target directory for active 3D volume uploads
├── download_data.py     # Script to fetch structural MNI test volumes
└── pyproject.toml       # Managed via uv

---

📦 Installation & Setup

1. Prerequisites

Ensure Ollama is installed and pull the model locally before starting the application:

ollama run llama3

---

2. Environment Initialization with uv

Create and activate a virtual environment, then install the required dependencies:

# Create a virtual environment
uv venv

# Activate the virtual environment

# macOS/Linux
source .venv/bin/activate

# Windows
.venv\Scripts\activate

# Install dependencies
uv pip install torch torchvision torchaudio
uv pip install monai nibabel langchain langchain-community fastapi uvicorn streamlit python-dotenv requests

---

3. Fetch Sample 3D Medical Data

Populate the staging area with a sample NIfTI (.nii.gz) brain volume:

python download_data.py

---

🏃 Running the Pipeline

The backend and frontend should run concurrently in separate terminals. Ensure the virtual environment is activated in both sessions.

Step A: Start the FastAPI Server

python backend/main.py

The API gateway will initialize at:

http://127.0.0.1:8000

---

Step B: Launch the Streamlit Dashboard

streamlit run frontend/app.py

The dashboard will be available at:

http://localhost:8501

---

🧪 Verifying the Execution Loop

1. Open the Streamlit UI at http://localhost:8501.
2. Upload the downloaded volume:

uploaded_scans/brain_sample.nii.gz

3. Submit the following query:

Run the 3D pipeline on this volume, extract the voxel matrix metrics, and map the high-intensity anomaly profiles against a historical target of 20.0 cc.

4. Click Run Agent Diagnostics.

5. Monitor the backend logs as the agent:

   • Parses the request
   • Invokes segmentation and analysis tools
   • Computes volumetric metrics
   • Generates a structured diagnostic summary

---

🔒 Privacy & Compliance

Neuro3D-Agent is designed for local execution and privacy-first deployments:

• All inference runs locally.
• No patient data is transmitted to external APIs.
• LLM interactions are handled through local Ollama-hosted models.
• Supports deployment strategies aligned with healthcare data governance requirements.

---

🎯 Example Workflow

Upload MRI Volume
        │
        ▼
FastAPI Backend
        │
        ▼
LangChain Agent
        │
 ┌──────┴──────┐
 ▼             ▼
MONAI      Ollama
Inference  Llama 3
 └──────┬──────┘
        ▼
Diagnostic Summary
        ▼
Streamlit Dashboard

---

📄 License

This project is intended for research, experimentation, and educational purposes. Clinical deployment should undergo appropriate validation, regulatory review, and institutional approval procedures.