|
1 | 1 | # Architecture Overview |
2 | 2 |
|
3 | 3 | Flamehaven FileSearch balances simplicity with production-grade safeguards. This |
4 | | -document describes the moving parts as of **v1.5.1**, featuring: |
| 4 | +document describes the moving parts as of **v1.5.2**, featuring: |
5 | 5 | - **Gravitas DSP Engine** (v1.3.1+) |
6 | 6 | - **Multimodal Search** (v1.4.0+) |
7 | 7 | - **pgvector with HNSW** (v1.4.0+) |
8 | 8 | - **Circuit Breaker & Health Checks** (v1.4.1+) |
9 | 9 | - **ABC base classes, ruff CI, Windows filename fix** (v1.4.2) |
10 | 10 | - **Universal Document Parser, Internal Chunker, Framework Integrations** (v1.5.0) |
11 | 11 | - **Dead code removal, critical complexity fixes, 360-test suite** (v1.5.1) |
| 12 | +- **Parse Cache, ContextExtractor, Backend Plugin Architecture** (v1.5.2) |
12 | 13 |
|
13 | 14 | --- |
14 | 15 |
|
@@ -85,7 +86,7 @@ The new **Chronos-Grid** integration handles high-speed vector storage and simil |
85 | 86 |
|
86 | 87 | ## 6. Testing & Quality (v1.4.2) |
87 | 88 |
|
88 | | -- **Test Framework**: `pytest` — 360 tests collected, all passing. |
| 89 | +- **Test Framework**: `pytest` — 443 tests collected, all passing (360 + 83 new). |
89 | 90 | - **Lint**: `black` (format) + `ruff` (lint/unused imports) — both enforced in CI. |
90 | 91 | - **Validation**: `validators.py` enforces security policies (Filename 200-char max, FileSize, SearchQuery XSS/SQLi checks). |
91 | 92 | - **SIDRCE Certification**: Omega 0.9894 (S++) — AI-Slop-Detector P0-P5 clean. |
@@ -118,19 +119,32 @@ The `engine/` sub-package contains the full parsing stack: |
118 | 119 |
|
119 | 120 | ``` |
120 | 121 | engine/ |
121 | | - file_parser.py — Dispatcher: routes by extension, tries parsers in priority order |
122 | | - format_parsers.py — Internal parsers: HTML, WebVTT, LaTeX, CSV, Image OCR |
123 | | - text_chunker.py — Structure-aware + token-aware RAG chunker (stdlib only) |
124 | | - embedding_generator.py — DSP v2.0 vectorizer |
125 | | - chronos_grid.py — Vector index + metadata store |
126 | | - gravitas_pack.py — Metadata compression |
127 | | - intent_refiner.py — Query analysis + search mode selection |
| 122 | + file_parser.py — Dispatcher: BackendRegistry.get(ext) -> backend.extract() |
| 123 | + format_backends.py — 11 AbstractFormatBackend classes + BackendRegistry (v1.5.2) |
| 124 | + format_parsers.py — Internal parsers: HTML, WebVTT, LaTeX, CSV, Image OCR |
| 125 | + parse_cache.py — mtime-based parse result cache (v1.5.2) |
| 126 | + context_extractor.py — RAG chunk context window extractor (v1.5.2) |
| 127 | + text_chunker.py — Structure-aware + token-aware RAG chunker (stdlib only) |
| 128 | + embedding_generator.py — DSP v2.0 vectorizer |
| 129 | + chronos_grid.py — Vector index + metadata store |
| 130 | + gravitas_pack.py — Metadata compression |
| 131 | + intent_refiner.py — Query analysis + search mode selection |
128 | 132 | ``` |
129 | 133 |
|
130 | | -**Extraction priority** (for each file): |
131 | | -1. Per-format internal parser (HTML/VTT/LaTeX/CSV via `format_parsers.py`) |
132 | | -2. Optional heavy parser (pymupdf, python-docx, openpyxl, python-pptx, striprtf) |
133 | | -3. Plain UTF-8 read (last resort for unknown text formats) |
| 134 | +**Extraction dispatch** (v1.5.2 — Backend Plugin): |
| 135 | + |
| 136 | +Each file extension resolves to an `AbstractFormatBackend` subclass via |
| 137 | +`BackendRegistry`. New formats plug in by subclassing and registering — no |
| 138 | +changes to `file_parser.py` required. |
| 139 | + |
| 140 | +**Extraction stack per backend:** |
| 141 | +- PDF: `PDFBackend` — pymupdf → pypdf fallback |
| 142 | +- DOCX/DOC: `DOCXBackend` / `DOCBackend` — python-docx + antiword |
| 143 | +- XLSX: `XLSXBackend` — openpyxl multi-sheet |
| 144 | +- PPTX: `PPTXBackend` — python-pptx text + tables |
| 145 | +- HTML/VTT/LaTeX/CSV: stdlib-only backends (zero extra deps) |
| 146 | +- Images: `ImageBackend` — pytesseract ([vision] extra) |
| 147 | +- Unknown: `PlainTextBackend` — UTF-8 fallback |
134 | 148 |
|
135 | 149 | **Content-based embedding** (v1.5.0): The first 2000 characters of extracted |
136 | 150 | content are used to generate the vector embedding, replacing the previous |
|
0 commit comments