feat: implement BM25 lexical search and hybrid retrieval capabilities

- Add a segmentation-free tokenizer for mixed Chinese/English text to support BM25 indexing.
- Introduce vector math helpers for dot product, norm, and cosine similarity calculations.
- Extend IVectorStore interface to include optional lexical search functionality.
- Implement LocalVectorStore to support BM25 indexing and hybrid retrieval.
- Update settings to include options for hybrid retrieval and MMR (Maximal Marginal Relevance) configurations.
- Add tests for BM25, tokenization, vector math, and hybrid recall mechanisms.
- Introduce a Vitest configuration for running tests in a Node environment with stubs for Obsidian.

Author: ssbeatty

CLAUDE.md

@@ -9,9 +9,19 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
 | `npm run dev` | Watch mode build via esbuild; outputs `main.js` |
 | `npm run build` | Type-check (`tsc --noEmit`) + production esbuild bundle (minified) |
 | `npm run lint` | Run ESLint (typescript-eslint + eslint-plugin-obsidianmd) |
+| `npm test` | Run the vitest unit suite (`vitest run`) |
+| `npm run test:watch` | Run vitest in watch mode |
 | `npm version <patch/minor/major>` | Bump version in `manifest.json` and `versions.json` |
 
-There are no unit tests. Testing is manual: build, copy `main.js` + `manifest.json` + `styles.css` to `<Vault>/.obsidian/plugins/obsidian-copilot/`, reload Obsidian, and enable the plugin.
+Unit tests live in `tests/` and run under vitest. They cover the dependency-free
+RAG primitives (tokenizer, BM25, RRF fusion, MMR, cosine math, chunk utilities)
+plus a synthetic `recall@k` benchmark (`tests/hybridRecall.test.ts`) that
+demonstrates hybrid + MMR retrieval beating pure top-K cosine. Tests only import
+modules that do **not** depend on `obsidian`, so they need no DOM/electron env.
+
+End-to-end testing is still manual: build, copy `main.js` + `manifest.json` +
+`styles.css` to `<Vault>/.obsidian/plugins/simple-rag-plugin/` (the `id` in
+`manifest.json`), reload Obsidian, and enable the plugin.
 
 ## High-Level Architecture
 
@@ -25,13 +35,16 @@ This is an Obsidian community plugin that adds an AI chat sidebar with RAG (Retr
    - Providers support a "thinking depth" (reasoning budget tokens) mapped via `ThinkingDepth`.
 
 2. **RAG Layer** (`src/rag/`)
-   - **Chunking**: Pluggable strategies — `DelimiterChunker`, `LengthChunker`, and `AIChunker` (uses an LLM call to semantically split text).
-   - **Embedding**: `EmbeddingService` calls an OpenAI-compatible `/embeddings` endpoint with batching.
+   - **Chunking**: Pluggable strategies resolved in `ChunkingResolver` — `DelimiterChunker`, `ParagraphChunker`, `LengthChunker`, `MarkdownHeadingChunker`, `MarkdownQAChunker`. `AIChunker` (LLM-based semantic split) exists but is **not** currently wired into the resolver — see the dead-code note below.
+   - **Embedding**: `EmbeddingService` calls an OpenAI-compatible `/embeddings` endpoint with batching, bounded concurrency, and retry/backoff (honoring `Retry-After`).
    - **Vector Store**: `IVectorStore` interface with two implementations:
-     - `LocalVectorStore` — persists to `.copilot/vectors.json` inside the vault.
-     - `RemoteVectorStore` — connects to Pinecone, Qdrant, or Weaviate.
+     - `LocalVectorStore` — persists to `.copilot/vectors.json` inside the vault; also serves BM25 lexical search (`lexicalSearch`) for hybrid retrieval.
+     - `PgvectorStore` — Postgres + the `pgvector` extension (cosine distance, HNSW index when available). Vector-only; no lexical search yet, so hybrid degrades gracefully to vector + MMR.
+   - **Retrieval primitives** (`src/rag/retrieval/`): dependency-free `tokenize` (CJK-bigram + Latin), `BM25`, `reciprocalRankFusion` (RRF), `mmrSelect` (MMR), and shared `cosineSimilarity`/vector math.
    - **Reranking**: `RerankService` supports Jina, Cohere, or custom rerank APIs.
-   - **Pipeline**: `RAGPipeline` orchestrates the flow: embed query → vector search → optional rerank → build context string.
+   - **Pipeline**: `RAGPipeline` orchestrates: embed query → vector over-fetch (threshold-filtered) → optional BM25 lexical search → RRF fusion → optional cross-encoder rerank → optional MMR diversification → neighbor-chunk expansion → build context string. Tunables live in `RAGConfig` (`hybrid`, `mmrEnabled`, `mmrLambda`, `neighborRadius`, `candidateMultiplier`).
+
+   > **Dead code**: `AIChunker` is implemented but unreferenced — it needs async chunking plumbing and an `AIService` dependency threaded through `createChunker`. Decide to wire it (as an `ai` strategy) or delete it; don't leave it dangling.
 
 3. **Indexing** (`src/indexer/VaultIndexer.ts`)
    - Incremental full-vault indexing that skips unchanged files by mtime.
@@ -43,16 +56,16 @@ This is an Obsidian community plugin that adds an AI chat sidebar with RAG (Retr
    - Two chat modes:
      - **Vault mode**: Uses `RAGPipeline.retrieve()` to inject relevant note chunks as context.
      - **Note mode**: Lets the user attach specific files or URLs; no vault search.
-   - Settings are split across four tabs: General, Models, Embedding, VectorDB.
+   - Settings are split across five tabs: General (incl. RAG + advanced retrieval), Models, Embedding, VectorDB, and LightRAG.
 
 ### Plugin Lifecycle (`src/main.ts`)
 
 `main.ts` should stay minimal. It:
 1. Loads persisted data (`settings` + `conversations`) via `loadData()`.
-2. Wires services together in `initServices()`: `AIService` → `EmbeddingService` + `RerankService` + `LocalVectorStore` + `IVectorStore` → `RAGPipeline` → `VaultIndexer`.
+2. Wires services together in `initServices()`: `AIService` → `EmbeddingService` + `RerankService` + (`LocalVectorStore` | `PgvectorStore`) as `IVectorStore` → `RAGPipeline` → `VaultIndexer`, plus `LightRAGClient` + `LightRAGSyncService`.
 3. Registers the `ChatView`, commands, and settings tab.
 4. On settings update, calls `reinitServices()`; if the vector store backend changed, it rebuilds the pipeline and indexer from scratch.
-5. On `onunload`, saves the local vector store and detaches the chat view.
+5. On `onunload`, saves the local vector store, stops LightRAG sync, closes the pgvector pool, and detaches the chat view.
 
 ### Key Patterns
 

eslint.config.mts

@@ -59,5 +59,9 @@ export default tseslint.config(
 		"version-bump.mjs",
 		"versions.json",
 		"main.js",
+		// Tests run under vitest (esbuild transpile), outside the src tsconfig
+		// project, so they're excluded from the typed lint pass.
+		"tests",
+		"vitest.config.mts",
 	]),
 );