v1.6.4 — Complexity Reduction Refactor

What's Changed

Refactored

• core.py — _restore_from_persistence (depth-5 / CC-17 → depth-3 / CC-5)
  Extracted three focused helpers:

  • _inject_into_chronos(uri, doc) — embedding generation + ChronosGrid injection
  • _restore_store_docs(store_name, docs) — doc dedup + restore loop
  • _restore_store_atoms(store_name, atoms) — atom restore loop
    Entry point reduced to 18 lines. No behavior change.

• auth.py — list_keys (depth-4 / CC-8 → depth-2)
  Extracted two static helpers:

  • _decode_permissions(perms_json) — decrypt + JSON parse with raw fallback
  • _row_to_key_info(row) — DB tuple → APIKeyInfo conversion
    Body reduced to a single list-comprehension.

• api.py — _init_searcher (depth-4 / CC-5 → depth-2)
  Extracted _seed_default_store(fs) — holds the nested inner-try that creates the default store and inserts the bootstrap doc.

Quality

• 71 tests pass (no regressions)
• Real-vault smoke probe: 156/156 files ingested, 8,405 atoms indexed, 5/5 queries status=success, confidence ≥ 0.840

Full Changelog: https://github.com/flamehaven01/Flamehaven-Filesearch/blob/main/CHANGELOG.md

[1.6.1] - 2026-04-19

Refactored

• API orchestration (api.py): initialize_services (66 lines, CC~8) →
  _init_searcher + _init_cache + _init_metrics + 8-line orchestrator.
  _record_upload_failure extracted — eliminates 2× duplicated
  record_file_upload + record_error blocks in upload_single_file.

• Admin auth (admin_routes.py): _get_admin_user (77 lines, CC~10) →
  _parse_bearer_token + _try_oauth_admin + _resolve_key_admin + 5-line
  orchestrator. Fixes reverse_field_glyphs rebuilt on every recursive call.

• Engine (engine/chronos_grid.py): seek_vector_resonance (80 lines,
  2 code paths) → _hnsw_vector_resonance + _brute_vector_resonance +
  10-line dispatcher (HNSW path vs brute-force cosine similarity).

• Engine (engine/gravitas_pack.py): _compress_dict / _decompress_dict
  clone cluster → _transform_dict(obj, key_map, value_transform) dispatch table.
  Both callers become 2-line delegators.

Changed

• eval_self.py: CORPUS_FILES split into AUDIT_CORPUS (11 docs) +
  SOURCE_CORPUS (7 source files). CORPUS_FILES = AUDIT_CORPUS + SOURCE_CORPUS
  preserves existing full-pack behaviour; AUDIT_CORPUS alone enables lightweight
  doc-quality runs.

• .gitignore: docs/history/ added under "Historical development artifacts".

Tests

• 475 passed, 13 skipped — same count as v1.6.0 (1 pre-existing flaky timing test
  in full suite; passes in isolation).

[1.6.0] - 2026-04-19

Added

• BM25 + RRF Hybrid Search (engine/hybrid_search.py): Production-grade BM25
  (k1=1.5, b=0.75) with Korean+English tokenizer
  (re.findall(r"[a-z0-9\uac00-\ud7a3]+", text.lower())).
  Reciprocal Rank Fusion merges BM25 and ChronosGrid semantic lists using
  string URI as doc ID — no integer alignment required. k=60, top_k configurable.
  Lazy per-store index with _bm25_dirty set: index rebuilt on first hybrid
  search after any upload, not on every upload.

• KnowledgeAtom chunk-level indexing (engine/knowledge_atom.py): Two-level
  indexing — file-level doc + chunk atoms with fragment URIs
  (local://store/enc_path#c0001). chunk_and_inject() splits content into
  800-char overlapping windows (120-char overlap, 80-char minimum), embeds each
  chunk via embedding_generator.generate(), injects into ChronosGrid, and
  registers in _atom_store_docs for URI-based resolution. Enables precision
  chunk-level retrieval alongside file-level documents.

• Stable URI scheme: Local documents now use
  local://<store>/<urllib.parse.quote(abs_path, safe='')> instead of
  local://<store>/<basename>. Eliminates collisions when files with identical
  names exist in different directories. URIs are reversible via unquote().
  Both main docs and chunk atoms share the same URI namespace.

Refactored

• core.py segmentation (1258 → 221 lines): FlamehavenFileSearch split into
  three focused mixin classes via IngestMixin, LocalSearchMixin,
  CloudSearchMixin. core.py is now a thin orchestrator: __init__,
  create_store, list_stores, delete_store, get_metrics,
  _resolve_vector_backend.

  Mixin	File	Responsibility
  IngestMixin	_ingest.py (228 L)	upload_file, upload_files, _local_upload, _generate_file_vector
  LocalSearchMixin	_search_local.py (273 L)	_local_search, BM25 rebuild, hybrid rerank, RAG prompt
  CloudSearchMixin	_search_cloud.py (265 L)	search, search_stream, search_multimodal + 6 shared helpers

• Duplicate helper elimination (_search_cloud.py): Six blocks that were
  copy-pasted between search() and search_multimodal() are now shared helpers:
  _resolve_search_params, _ensure_store, _query_vector_backend,
  _driftlock_validate, _extract_grounding_sources, _gemini_search_call.

Fixed

• search_stream double intent-refine bug: intent_refiner.refine_intent(query)
  was called twice (lines 984 and 988 in old core.py) — once before the
  provider-RAG branch and once inside it. The second call discarded the first
  optimized_query. Fixed: single call, result reused throughout the method.

Tests

• 443 tests pass, 13 skipped — no regression from refactor.
• test_flamehaven_remote_client_flow patch target updated: also patches
  flamehaven_filesearch._search_cloud._google_genai_types after types moved
  from core.py to _search_cloud.py.

[1.5.3] - 2026-04-19

[1.5.3] - 2026-04-19

Added

• Multi-provider LLM support (engine/llm_providers.py): AbstractLLMProvider
  ABC + 4 concrete implementations + create_llm_provider() factory.

  Provider	Class	Install extra
  Google Gemini	GeminiProvider	[google] (existing)
  OpenAI ChatGPT	OpenAIProvider	[openai]
  Anthropic Claude	AnthropicProvider	[anthropic]
  Ollama local	OllamaProvider	[ollama]
  OpenAI-compatible	OpenAIProvider + base_url	[openai]

• Local model support via Ollama (OllamaProvider): zero API key required.
  Tested models: gemma4:27b, gemma4:4b, gemma4:2b (128K/256K ctx, Apache-2.0),
  qwen2.5:7b/14b/32b, mistral, llama3.2. Streaming via /api/generate.

• OpenAI-compatible endpoint routing: openai_compatible / kimi / vllm /
  lmstudio all map to OpenAIProvider with custom base_url.
  Example: Kimi — OPENAI_BASE_URL=https://api.moonshot.cn/v1.

• Provider-RAG mode in core.py: for non-Gemini providers, search() runs
  local semantic retrieval (ChronosGrid) → _build_rag_prompt() → LLM answer.
  search_stream() calls provider.stream() for token-by-token output.

• New Config fields:

  • llm_provider (str, default "gemini")
  • openai_api_key, anthropic_api_key (auto-loaded from env)
  • ollama_base_url (default http://localhost:11434)
  • local_model (default gemma4:27b)
  • openai_base_url (for compatible endpoints)

• New env vars: LLM_PROVIDER, OPENAI_API_KEY, ANTHROPIC_API_KEY,
  OLLAMA_BASE_URL, LOCAL_MODEL, OPENAI_BASE_URL

• New install extras: [openai], [anthropic], [ollama]

Changed

• Config.validate() skips Google API key requirement when llm_provider != "gemini"
• pyproject.toml: description updated; keywords extended with openai,
  anthropic, claude, ollama, gemma, qwen, local-llm

[1.5.2] - 2026-04-19

Added

• Parse Cache (engine/parse_cache.py): mtime-based file extraction cache.
  Algorithm absorbed from RAG-Anything processor.py:_generate_cache_key().
  Cache key = MD5(resolved_path + mtime + parser_config). Path-indexed reverse
  map enables O(1) invalidate(). API: get/put/invalidate/clear/stats.
  extract_text(use_cache=True) integrates transparently — no API change.
  Score: 0.0 CLEAN.

• ContextExtractor (engine/context_extractor.py): Sliding-window chunk
  context extractor for RAG result enrichment. Algorithm absorbed from
  RAG-Anything modalprocessors.py:ContextExtractor. Given chunk_text()
  output, enrich_chunks() adds a context key to each chunk containing
  surrounding neighbour text. ContextConfig: window_size, max_context_chars,
  include_headings. Zero external dependencies. Score: 0.0 CLEAN.

• Backend Plugin Architecture (engine/format_backends.py): Format-family
  backends absorbed from Docling abstract_backend.py pattern.
  AbstractFormatBackend ABC with supported_extensions + extract().
  BackendRegistry maps extensions to backend classes; new formats register
  without modifying the dispatcher. 11 concrete backends:
  PDFBackend, DOCXBackend, DOCBackend, XLSXBackend, PPTXBackend,
  RTFBackend, HTMLBackend, VTTBackend, LaTeXBackend, CSVBackend,
  ImageBackend, PlainTextBackend. Score: 12.2 clean.

Refactored

• engine/file_parser.py (75 lines, was 340): Rewritten as pure registry
  dispatcher — _dispatch() resolves backend via BackendRegistry.get(ext)
  then calls backend.extract(). Cyclomatic complexity 13 → 3.
  function_clone_cluster (5 structurally similar _extract_* functions)
  eliminated by moving each into its own Backend class. Score: 3.0 CLEAN.

Tests

• tests/test_phase1_parse_cache_context.py: 33 tests (parse_cache + ContextExtractor).
• tests/test_phase2_format_backends.py: 50 tests (registry + backends + helpers).
• Combined: 83 tests, all passing. AI-Slop-Detector critical deficits: 0.

[1.5.1] - 2026-04-18

Removed

• engine/embedding_generator_legacy.py: Deleted. Identical API surface and
  100% function overlap with embedding_generator.py; the file was never imported
  by production code and represented 306 lines of dead duplicate code.

Refactored

• engine/text_chunker.py: Extracted _split_section() and _make_chunk()
  helpers from chunk_text() to reduce nesting depth and cyclomatic complexity.

• engine/file_parser.py: Extracted _extract_table_rows() from
  _extract_pptx() (nesting depth 5 → 3); split bare except clauses in
  _extract_doc() into typed FileNotFoundError / subprocess.TimeoutExpired
  handlers with proper log messages.

• engine/gravitas_pack.py: Extracted _estimate_field_reduction() from
  estimate_compression_ratio() to eliminate nested for-loops.

• engine/chronos_grid.py: Extracted _upsert_vector() from
  inject_essence() to collapse 4-level nesting.

• usage_middleware.py: Extracted _collect_exceeded_quotas() module-level
  helper to flatten nested loop inside dispatch().

• api.py: Extracted _save_upload_file() from upload_multiple_files()
  to resolve critical nested_complexity (depth=4 + high cyclomatic complexity).

Tests

• 360 tests pass (13 skipped) — 29 more than v1.5.0 (360 vs 331).
• AI-Slop-Detector: CLEAN | critical deficits 7 → 0 | avg deficit score
  13.46 → 11.25.
• ruff: no issues.

[1.5.0] - 2026-04-16

Added

• Universal Document Parser (engine/file_parser.py): Complete rewrite with
  support for 34 file extensions across 10 format families. Extraction stack:
  PDF (pymupdf → pypdf), DOCX/DOC (python-docx + antiword), XLSX (openpyxl),
  PPTX (python-pptx), RTF (striprtf), HTML, WebVTT, LaTeX, CSV (all stdlib),
  Image OCR ([vision] extra).

• Internal Format Parsers (engine/format_parsers.py): Zero-dependency
  implementations absorbed directly into the codebase — no external document-AI
  framework required:

  • HTML (extract_html): stdlib html.parser-based extractor; suppresses
    <script>, <style>, <head> content; preserves block structure.
  • WebVTT (extract_vtt): W3C WebVTT spec regex parser; strips timestamps,
    cue settings, NOTE/STYLE/REGION blocks, and inline tags (<b>, <c.*>).
  • LaTeX (extract_latex): Regex-based text extraction; removes display math
    and figure environments, promotes \section headings, unwraps \textbf{} /
    \emph{} etc., strips remaining commands.
  • CSV (extract_csv): stdlib csv.Sniffer with auto-detected delimiter
    (,, ;, TAB, |, :); fallback to comma on detection failure.
  • Image OCR (extract_image): Delegates to pytesseract when [vision] extra
    is installed; gracefully returns empty string otherwise.

• Internal Text Chunker (engine/text_chunker.py): Structure-aware + token-
  aware chunking for RAG pipelines — no external ML dependency:

  • Phase 1: Markdown heading boundary splitting (heading stack preserved).
  • Phase 2: Paragraph splitting within sections; sentence splitting for
    oversized paragraphs.
  • Phase 3: Undersized chunk merging (merge_peers).
  • Token estimate: 1 token ≈ 0.75 words (conservative for embedding models).
  • API: chunk_text(text, max_tokens=512, min_tokens=64, merge_peers=True)
    → List[{text, pages, headings}].

• Framework Integrations (integrations/): Plug-and-play adapters for
  popular AI agent frameworks — all built on internal extraction, no third-party
  document-AI required:

  • FlamehavenLangChainLoader — LangChain BaseLoader interface; supports
    chunk=True for node-level splits.
  • FlamehavenLlamaIndexReader — LlamaIndex BaseReader interface; supports
    chunk=True.
  • FlamehavenHaystackConverter — Haystack BaseConverter interface;
    run(sources=[...]) returns {"documents": [...]}.
  • FlamehavenCrewAITool — CrewAI BaseTool interface; _run() and
    _arun() for sync and async agents.

• Content-Based Vector Embeddings: Upload pipeline now extracts file content
  (first 2000 chars) and embeds it via DSP v2.0. Previously, embeddings were
  generated from filename + filetype strings, making semantic search meaningless
  for local mode. Fixes semantic search quality for all non-Gemini-API paths.

Changed

• engine/file_parser.py: Fully rewritten. Docling external dependency
  removed; all parsers are now internal or delegate to existing [parsers] extras.
  SUPPORTED_EXTENSIONS expanded from 11 to 34 entries.

• pyproject.toml: packages list explicitly includes
  flamehaven_filesearch.engine and flamehaven_filesearch.integrations
  sub-packages for correct PyPI distribution.

• validators.py: Removed HWP MIME types (application/x-hwp,
  application/haansofthwp, application/vnd.hancom.hwp/hwpx). Added audio
  (audio/wav, audio/mpeg, audio/ogg, audio/flac, audio/aac,
  audio/x-m4a), WebVTT (text/vtt), and LaTeX (application/x-latex,
  text/x-tex) MIME types.

Removed

• HWP / HWPX support: The OLE binary HWP parser (_extract_hwp,
  _parse_hwp5_body) and HWPX ZIP+XML parser have been removed. HWP requires
  the olefile dependency and a custom binary record parser that adds
  significant complexity for a narrow format. Use .docx conversion instead.

Tests

• 318 tests pass (13 skipped). All format parser functions validated with
  tempfile-based unit checks.
• AI-Slop-Detector: status CLEAN, all new files LDR S++, inflation PASS.
• ruff: no issues across all new and modified files.

[1.4.2] - 2026-04-16

Changed

• CI/CD: Replaced flake8 with ruff in the lint job for faster and
  consistent linting (matches local development tooling).
• Abstract base classes: VectorStore, MetadataStore, and IAMProvider
  migrated from raise NotImplementedError stubs to proper ABC +
  @abstractmethod — cleaner Python contract, eliminates unreachable
  NotImplementedError paths.
• tokenize() in lang_processor.py: Removed implicit detect_language()
  call; callers must pass lang explicitly. Removes a hidden latency source
  and makes call-site intent clear.

Added

• NullIAMProvider: Concrete null-object implementation of IAMProvider
  for use when no IAM backend is configured (replaces direct abstract
  instantiation).
• .slopconfig.yaml: Project-specific AI-Slop-Detector configuration —
  suppresses false positives for optional dependencies, ABC stubs, and
  FastAPI singleton globals; adds Flamehaven-specific domain overrides.
• _xlsx_row_text() / _pptx_table_lines(): Extracted helpers in
  file_parser.py to reduce nesting depth and improve readability.

Fixed

• MAX_FILENAME_LENGTH 255 → 200 (validators.py): Prevents Windows
  MAX_PATH (260 chars) overflow when writing uploaded files to temp
  directories, fixing test_very_long_filename → 500 regression.
• Logging fallback JSON output: CustomJsonFormatter (when
  python-json-logger is absent) now emits proper JSON instead of plain text,
  fixing JSONDecodeError in test_setup_json_logging_accepts_level_kwarg.
• setup_json_logging() else branch: Used logging.Formatter() instead of
  CustomJsonFormatter() — corrected to CustomJsonFormatter().
• Vector generation latency (embedding_generator.py): Added ASCII
  shortcut — skips detect_language() for ASCII-only text, reducing avg
  generation time from 14.9 ms to 0.847 ms (p95 < 1 ms).
• Empty except blocks: Converted silent exception swallowing in
  ws_routes.py, multimodal.py, and vector_store.py to logger.debug()
  calls with the captured exception.
• Unused imports: Removed from typing import Generator (inline,
  core.py) and HTTPException, Response, status (usage_middleware.py).

Tests

• 331 tests collected, all pass under pytest.
• test_very_long_filename: now correctly returns 400 (not 500) on Windows.
• test_performance_*: avg vector generation < 1 ms (threshold met).
• test_setup_json_logging_*: fallback JSON formatter validated.

v1.4.1 - Usage Tracking & pgvector Hardening

Production-hardening release with comprehensive usage tracking, quota management, and pgvector reliability enhancements.

Full Changelog: CHANGELOG.md

Key Features

Usage Tracking & Quota Management

• Per-API-key request/token tracking with SQLite backend
• Daily and monthly quota enforcement
• Alert system with configurable thresholds
• Automatic cleanup of old records

Admin APIs

• Detailed usage statistics
• Quota status and configuration
• Usage alerts monitoring

pgvector Maintenance

• HNSW index rebuilding
• VACUUM ANALYZE operations
• Index statistics export
• Comprehensive tuning guide

Installation

pip install flamehaven-filesearch==1.4.1

Upgrade

pip install --upgrade flamehaven-filesearch

Documentation

• CHANGELOG.md
• pgvector Tuning Guide
• Implementation Summary

v1.3.1

Added - Phase 2 & 3: Semantic Search + Gravitas-Pack Integration

• Gravitas Vectorizer v2.0: Custom Deterministic Semantic Projection (DSP) algorithm
  • Zero ML dependencies (removed sentence-transformers 500MB+)
  • Instant initialization (<1ms vs 2min+ before)
  • Hybrid feature extraction: word tokens (2.0x weight) + char n-grams (3-5)
  • Signed feature hashing for collision mitigation
  • 384-dimensional unit-normalized vectors
  • LRU caching with 16.7%+ hit rate
• GravitasPacker: Symbolic compression integrated into cache layer
  • 90%+ compression ratio on metadata
  • Deterministic lore scroll generation
  • Instant decompression (<1ms)
• Vector Quantizer: int8 quantization for 75% memory reduction
  • Asymmetric quantization (per-vector min/max calibration)
  • 30%+ speedup on cosine similarity calculations
  • Backward compatible with float32 vectors
• Search Modes: keyword, semantic, hybrid via search_mode parameter
• API Schema Enhancement: Added refined_query, corrections, search_mode, search_intent, semantic_results to SearchResponse
• Chronos-Grid Integration: Vector storage and similarity search
• Intent-Refiner: Typo correction and query optimization
• unittest Migration: Replaced pytest with Python stdlib unittest
  • 19/19 tests passing in 0.33s
  • Zero timeout issues
  • Master test suite: tests/run_all_tests.py
• Performance Benchmarks: Dedicated benchmark suite for DSP v2.0
• Documentation: README, CHANGELOG, TOMB files updated

Performance

• Vector generation: <1ms per text
• Memory: 1536 bytes → 384 bytes per vector (75% reduction)
• Metadata compression: 90%+ ratio
• Search speed: 30% faster on quantized vectors
• Similar text similarity: 0.787 (78.7%)
• Differentiation ratio: 2.36x
• Cache efficiency: 16.7%+ hit rate
• Search 1000 vectors: <100ms
• Precision loss: <0.1% (negligible for file search)

Changed

• EmbeddingGenerator completely rewritten with DSP algorithm
• cache.py: Integrated GravitasPacker compression/decompression
• chronos_grid.py: Optional quantization support
• Test infrastructure migrated from pytest to unittest
• Docker metadata updated to v1.3.1
• README badges and features updated
• Version bumped to 1.3.1

Fixed

• Critical: pytest timeout issue resolved via unittest migration
• Critical: sentence-transformers blocking eliminated
• ASCII safety enforced across all output (no Unicode in cp949 environment)

Tests

• python tests/run_all_tests.py (19/19 passed, 0.33s)
• All semantic search features verified
• Performance benchmarks included

Breaking Changes

• None (fully backward compatible)
• search_mode parameter is optional with default behavior preserved