|
| 1 | +# OpenDocuments Architecture |
| 2 | + |
| 3 | +OpenDocuments is a self-hosted RAG platform for searching private documents across GitHub, Notion, Google Drive, local files, and other sources. |
| 4 | + |
| 5 | +The system is designed around three principles: |
| 6 | + |
| 7 | +- **Core-first design**: business logic lives in `@opendocuments/core`, while CLI, server, web, and SDK layers reuse it. |
| 8 | +- **Plugin-first extension**: connectors, parsers, and model providers are independent plugins. |
| 9 | +- **Source-grounded retrieval**: answers are generated from retrieved document context and returned with source references. |
| 10 | + |
| 11 | +## System Overview |
| 12 | + |
| 13 | +```mermaid |
| 14 | +flowchart LR |
| 15 | + subgraph Sources["Document Sources"] |
| 16 | + GH["GitHub"] |
| 17 | + NO["Notion"] |
| 18 | + GD["Google Drive"] |
| 19 | + LF["Local Files"] |
| 20 | + WS["Web / S3 / Confluence"] |
| 21 | + end |
| 22 | +
|
| 23 | + subgraph Plugins["Plugin Layer"] |
| 24 | + CON["Connector Plugins"] |
| 25 | + PAR["Parser Plugins"] |
| 26 | + MOD["Model Provider Plugins"] |
| 27 | + end |
| 28 | +
|
| 29 | + subgraph Core["@opendocuments/core"] |
| 30 | + ING["Ingest Pipeline"] |
| 31 | + RAG["RAG Engine"] |
| 32 | + AUTH["Auth / Security"] |
| 33 | + BUS["Typed Event Bus"] |
| 34 | + end |
| 35 | +
|
| 36 | + subgraph Storage["Storage"] |
| 37 | + SQL[("SQLite\nmetadata + FTS5")] |
| 38 | + VEC[("LanceDB\nvectors")] |
| 39 | + end |
| 40 | +
|
| 41 | + subgraph Interfaces["Interfaces"] |
| 42 | + CLI["CLI"] |
| 43 | + API["HTTP API"] |
| 44 | + WEB["Web UI"] |
| 45 | + SDK["TypeScript SDK"] |
| 46 | + MCP["MCP Server"] |
| 47 | + end |
| 48 | +
|
| 49 | + GH --> CON |
| 50 | + NO --> CON |
| 51 | + GD --> CON |
| 52 | + LF --> CON |
| 53 | + WS --> CON |
| 54 | + CON --> ING |
| 55 | + ING --> PAR |
| 56 | + ING --> SQL |
| 57 | + ING --> VEC |
| 58 | + CLI --> RAG |
| 59 | + API --> RAG |
| 60 | + WEB --> RAG |
| 61 | + SDK --> RAG |
| 62 | + MCP --> RAG |
| 63 | + RAG --> SQL |
| 64 | + RAG --> VEC |
| 65 | + RAG --> MOD |
| 66 | + RAG --> API |
| 67 | + RAG --> CLI |
| 68 | + AUTH --> API |
| 69 | + BUS --> ING |
| 70 | + BUS --> RAG |
| 71 | +``` |
| 72 | + |
| 73 | +## Monorepo Layout |
| 74 | + |
| 75 | +```mermaid |
| 76 | +flowchart TB |
| 77 | + ROOT["OpenDocuments Monorepo"] |
| 78 | +
|
| 79 | + ROOT --> PKG["packages/"] |
| 80 | + ROOT --> PLG["plugins/"] |
| 81 | + ROOT --> DOCS["docs-site/"] |
| 82 | +
|
| 83 | + PKG --> CORE["core\nRAG, ingest, storage, auth, plugins"] |
| 84 | + PKG --> SERVER["server\nHono HTTP API, MCP server"] |
| 85 | + PKG --> CLI["cli\nCommander.js commands"] |
| 86 | + PKG --> WEB["web\nReact + Vite UI"] |
| 87 | + PKG --> CLIENT["client\nTypeScript SDK"] |
| 88 | +
|
| 89 | + PLG --> MODEL["model-*\nOllama, OpenAI, Anthropic, Google, Grok"] |
| 90 | + PLG --> PARSER["parser-*\nPDF, DOCX, XLSX, HTML, Code, PPTX"] |
| 91 | + PLG --> CONNECTOR["connector-*\nGitHub, Notion, GDrive, S3, Web"] |
| 92 | +
|
| 93 | + DOCS --> SITE["VitePress documentation"] |
| 94 | +``` |
| 95 | + |
| 96 | +`@opendocuments/core` is the center of the architecture. It exposes reusable APIs for ingest, retrieval, storage, authentication, and plugins. The outer packages are protocol or user-interface layers. |
| 97 | + |
| 98 | +## Ingest Pipeline |
| 99 | + |
| 100 | +The ingest pipeline converts external documents into searchable metadata and embeddings. |
| 101 | + |
| 102 | +```mermaid |
| 103 | +sequenceDiagram |
| 104 | + participant Source as Document Source |
| 105 | + participant Connector as Connector Plugin |
| 106 | + participant Parser as Parser Plugin |
| 107 | + participant Ingest as Ingest Pipeline |
| 108 | + participant Model as Embedding Provider |
| 109 | + participant SQLite as SQLite / FTS5 |
| 110 | + participant LanceDB as LanceDB |
| 111 | +
|
| 112 | + Source->>Connector: discover and fetch documents |
| 113 | + Connector->>Parser: pass file or raw content |
| 114 | + Parser->>Ingest: normalized document text |
| 115 | + Ingest->>Ingest: structure-aware chunking |
| 116 | + Ingest->>Model: generate embeddings |
| 117 | + Ingest->>SQLite: store metadata and keyword index |
| 118 | + Ingest->>LanceDB: store vectors and chunk payloads |
| 119 | +``` |
| 120 | + |
| 121 | +Key responsibilities: |
| 122 | + |
| 123 | +- Normalize documents from different sources into a common document model. |
| 124 | +- Preserve useful structure such as headings, sections, and code blocks. |
| 125 | +- Store metadata in SQLite and vector payloads in LanceDB. |
| 126 | +- Keep parser and connector logic outside the core pipeline through plugins. |
| 127 | + |
| 128 | +## RAG Pipeline |
| 129 | + |
| 130 | +OpenDocuments uses a retrieval pipeline that combines semantic search, keyword search, query expansion, reranking, and grounding checks. |
| 131 | + |
| 132 | +```mermaid |
| 133 | +flowchart LR |
| 134 | + Q["User Question"] --> INT["Intent Classification"] |
| 135 | + INT --> EXP["Query Expansion\nHyDE / Multi-query"] |
| 136 | + EXP --> DENSE["Vector Search\nLanceDB"] |
| 137 | + EXP --> SPARSE["Keyword Search\nSQLite FTS5"] |
| 138 | + DENSE --> RRF["RRF Merge"] |
| 139 | + SPARSE --> RRF |
| 140 | + RRF --> PARENT["Parent Context\nDocument Expansion"] |
| 141 | + PARENT --> RERANK["Rerank Results"] |
| 142 | + RERANK --> FIT["Context Fitting"] |
| 143 | + FIT --> GEN["Answer Generation"] |
| 144 | + GEN --> GROUND["Grounding Check"] |
| 145 | + GROUND --> OUT["Answer + Sources\nConfidence"] |
| 146 | +``` |
| 147 | + |
| 148 | +Important retrieval features: |
| 149 | + |
| 150 | +- **Hybrid search**: combines dense vector search with SQLite FTS5 keyword search. |
| 151 | +- **RRF merge**: merges dense and sparse search results with Reciprocal Rank Fusion. |
| 152 | +- **HyDE and multi-query**: expands difficult questions into better retrieval queries. |
| 153 | +- **Parent document retrieval**: restores broader section context around matching chunks. |
| 154 | +- **Reranking**: improves final context selection before generation. |
| 155 | +- **Grounding check**: verifies that generated answers are supported by retrieved sources. |
| 156 | + |
| 157 | +## Plugin Architecture |
| 158 | + |
| 159 | +Plugins let OpenDocuments add new models, document formats, and external sources without changing the core RAG pipeline. |
| 160 | + |
| 161 | +```mermaid |
| 162 | +flowchart TB |
| 163 | + subgraph Core["Core Plugin Contracts"] |
| 164 | + C["ConnectorPlugin"] |
| 165 | + P["ParserPlugin"] |
| 166 | + M["ModelPlugin"] |
| 167 | + end |
| 168 | +
|
| 169 | + subgraph Connectors["Connector Plugins"] |
| 170 | + CGH["GitHub"] |
| 171 | + CNO["Notion"] |
| 172 | + CGD["Google Drive"] |
| 173 | + CWEB["Web Crawler / Search"] |
| 174 | + end |
| 175 | +
|
| 176 | + subgraph Parsers["Parser Plugins"] |
| 177 | + PPDF["PDF"] |
| 178 | + PDOCX["DOCX"] |
| 179 | + PXLSX["XLSX"] |
| 180 | + PCODE["Code"] |
| 181 | + PPPTX["PPTX"] |
| 182 | + end |
| 183 | +
|
| 184 | + subgraph Models["Model Provider Plugins"] |
| 185 | + MO["Ollama"] |
| 186 | + MOP["OpenAI"] |
| 187 | + MA["Anthropic"] |
| 188 | + MG["Google"] |
| 189 | + MX["Grok"] |
| 190 | + end |
| 191 | +
|
| 192 | + C --> CGH |
| 193 | + C --> CNO |
| 194 | + C --> CGD |
| 195 | + C --> CWEB |
| 196 | + P --> PPDF |
| 197 | + P --> PDOCX |
| 198 | + P --> PXLSX |
| 199 | + P --> PCODE |
| 200 | + P --> PPPTX |
| 201 | + M --> MO |
| 202 | + M --> MOP |
| 203 | + M --> MA |
| 204 | + M --> MG |
| 205 | + M --> MX |
| 206 | +``` |
| 207 | + |
| 208 | +This keeps the core package focused on orchestration and contracts, while plugin packages own provider-specific behavior. |
| 209 | + |
| 210 | +## Storage Design |
| 211 | + |
| 212 | +OpenDocuments uses two storage layers because metadata search and vector search have different access patterns. |
| 213 | + |
| 214 | +| Layer | Technology | Purpose | |
| 215 | +| --- | --- | --- | |
| 216 | +| Metadata store | SQLite | workspaces, documents, chunks, jobs, auth data | |
| 217 | +| Keyword index | SQLite FTS5 | sparse keyword search and exact-match retrieval | |
| 218 | +| Vector store | LanceDB | embeddings and semantic similarity search | |
| 219 | + |
| 220 | +This design keeps local self-hosted setup simple while preserving a clear path to swap storage implementations later. |
| 221 | + |
| 222 | +## Interface Layers |
| 223 | + |
| 224 | +```mermaid |
| 225 | +flowchart LR |
| 226 | + CORE["@opendocuments/core"] |
| 227 | +
|
| 228 | + CLI["CLI\nopendocuments"] |
| 229 | + SERVER["Server\nHono HTTP API"] |
| 230 | + WEB["Web UI\nReact + Vite"] |
| 231 | + CLIENT["Client SDK\nTypeScript"] |
| 232 | + MCP["MCP Server\nAI assistant access"] |
| 233 | +
|
| 234 | + CLI --> CORE |
| 235 | + SERVER --> CORE |
| 236 | + WEB --> SERVER |
| 237 | + CLIENT --> SERVER |
| 238 | + MCP --> CORE |
| 239 | +``` |
| 240 | + |
| 241 | +Interface layers are intentionally thin: |
| 242 | + |
| 243 | +- CLI exposes local commands for setup, indexing, asking, diagnostics, and backup. |
| 244 | +- Server exposes HTTP APIs, authentication middleware, MCP server, and widget endpoints. |
| 245 | +- Web UI consumes server APIs for browser-based operation. |
| 246 | +- TypeScript SDK gives external applications a typed API client. |
| 247 | +- MCP server exposes the knowledge base to AI coding assistants. |
| 248 | + |
| 249 | +## Security Considerations |
| 250 | + |
| 251 | +Security-sensitive paths are handled close to the storage, server, and query layers: |
| 252 | + |
| 253 | +- SQL queries use parameterized statements. |
| 254 | +- SQLite FTS5 queries are escaped before execution. |
| 255 | +- LanceDB filters are built through safe where-clause helpers. |
| 256 | +- Team mode endpoints are protected by authentication middleware. |
| 257 | +- Error responses avoid leaking stack traces or internal paths in production. |
| 258 | + |
| 259 | +## Design Tradeoffs |
| 260 | + |
| 261 | +| Decision | Why | |
| 262 | +| --- | --- | |
| 263 | +| SQLite + LanceDB | Simple self-hosted setup with separate metadata and vector search layers | |
| 264 | +| Plugin-first architecture | New sources, parsers, and model providers can be added without changing core | |
| 265 | +| Core-first monorepo | CLI, server, web, SDK, and MCP reuse the same business logic | |
| 266 | +| Hono server layer | Lightweight TypeScript-friendly HTTP layer around core services | |
| 267 | +| Retrieval profiles | Users can trade speed for quality with fast, balanced, and precise modes | |
0 commit comments