MCP tools hang forever after an interrupted mine — chromadb 1.x compactor cannot backfill WAL into the HNSW segment (CLI unaffected)

[KeilerHirsch]

Summary

After a mempalace mine is interrupted/killed mid-write, every MCP tool call hangs indefinitely (mempalace_status, mempalace_search, mempalace_reconnect, mempalace_add_drawer) — no timeout, no error. The CLI keeps working (mempalace status, mempalace repair-status return in seconds). The hang survives full reboots because the broken state is on disk. Root cause is a damaged HNSW segment writer + pending rows in chromadb's embeddings_queue that the chromadb-1.x compactor cannot apply.

Environment

• mempalace 3.4.0, chromadb 1.5.9 (Rust core / "compactor")
• Windows 11, Python 3.13, onnxruntime-gpu, embedding model embeddinggemma (384-dim), backend chroma
• Palace size ~268k drawers (~1.4 GB sqlite + ~450 MB HNSW)

Symptom

• MCP: any tool that opens the chroma collection hangs forever. The harness logs Tool 'mempalace_status' still running (720s elapsed). The hang is in native code — it ignores SIGINT.
• CLI: mempalace status and mempalace repair-status work fine (they read sqlite counts and never open a chromadb client).

Root cause

An interrupted mine leaves unflushed rows in chroma's embeddings_queue plus a damaged HNSW segment writer. Any operation that opens the collection (count(), query, add) makes the chromadb-1.x compactor try to backfill the queue into the HNSW segment, which fails:

chromadb.errors.InternalError: Error executing plan:
  Error sending backfill request to compactor:
  Failed to apply logs to the hnsw segment writer

In the long-lived MCP server this surfaces as an infinite hang; called directly from Python it raises the error above.

How to confirm (read-only)

import chromadb
c = chromadb.PersistentClient(path=r"<HOME>/.mempalace/palace")
for col in c.list_collections():
    print(col.name, col.count())   # -> InternalError: ...hnsw segment writer

Note: mempalace repair-status passes (sqlite vs HNSW counts match within flush-lag) because it never opens the chroma client — so it does not catch a broken segment writer.

What it was NOT (ruled out, each with evidence)

• Stale lock files — cleaned ~/.mempalace/locks/mine_palace_*.lock + hook_state/mine_pids/*.pid; still hangs. (Separate leak: 2200+ empty locks/*.lock accumulate.)
• HNSW corruption — repair-status OK; link_lists.bin tiny (~2 MB), no blowup.
• HF Hub network — model is cached locally; HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 still hangs.
• GPU/CUDA/TensorRT — embedding_device: cpu still hangs (the hang is before embedding).
• max_seq_id poison — values sane (max_seq_id == embeddings count).

Fix that worked

repair --mode from-sqlite rebuilds the HNSW from the sqlite ground truth, bypassing the broken client/segment:

# In-place fails on Windows (see gotcha below), so rebuild into a new path
# reading the live sqlite read-only:
mempalace --palace <HOME>/.mempalace/palace_rebuilt repair \
          --mode from-sqlite --source <HOME>/.mempalace/palace --yes
mempalace --palace <HOME>/.mempalace/palace_rebuilt palace set-embedder --model embeddinggemma
# then point config.json "palace_path" at palace_rebuilt and restart the MCP server

Result: rebuilt 269,491 rows; count() OK, repair-status divergence 0, end-to-end MCP status/search/add_drawer work again.

Windows gotcha

In-place repair --mode from-sqlite --archive-existing fails with:

PermissionError: [WinError 32] ... data_level0.bin is used by another process

because the running MCP server holds the HNSW segment files open, so the archive-rename can't proceed. Either stop the MCP server first, or rebuild into a separate path from the live sqlite (read-only) as above.

Suggested hardening

1. Fail fast, don't hang. Wrap collection-open / compaction with an operation deadline (and a SQLite busy_timeout) so a stuck compactor returns a structured error instead of hanging the whole MCP server forever.
2. Surface the real cause. Detect the Failed to apply logs to the hnsw segment writer compactor error and point the user to repair --mode from-sqlite instead of an opaque infinite hang.
3. mine crash-safety. Make queue/WAL application atomic or recoverable so an interrupted mine can't leave an unbackfillable segment; on startup, detect a dirty queue and offer a guarded rebuild.
4. Lock hygiene. mine_palace_lock (palace.py ~L945–984) never liveness-checks the recorded holder PID (no os.kill(pid,0) / lease) and leaks empty locks/*.lock. Add PID-liveness/lease + cleanup-on-release.
5. Deeper repair-status. Since it never opens the chroma client, it can't see a broken segment writer — consider an optional guarded count() probe so the broken state is detectable without a 6-hour investigation.

[mvalentsev]

This is the chromadb 1.5.x HNSW-corruption cluster, and specifically a re-occurrence of #1296 (closed): a partial or interrupted mine leaves rows in embeddings_queue, the segment then refuses to re-apply, and count() fails with the same "Error sending backfill request to compactor". #1296's recovery and detection were handled (#1310 added repair --mode from-sqlite, #1285 added the preflight quarantine), but the prevention half never landed.

• Detection / fail-fast (your points 1 and 5): fix: harden search and repair against transient index corruption #1598 (open) adds a subprocess-isolated loadability probe. python -m mempalace._loadprobe <palace> opens the collection and counts in a fresh process with a timeout (default 120s), called from the status/search/MCP-startup paths via validate_palace_health. A torn or wedged segment surfaces as "not loadable, run repair --mode from-sqlite" instead of hanging or segfaulting the host. Being subprocess + timeout based, it bounds the write-side hang here, not only the read-side crash the other reports hit.
• Crash-safety / atomic queue+WAL (your point 3): the interrupted/concurrent-writer prevention side, tracked in HNSW corruption from concurrent ChromaDB clients (MCP server + hook subprocesses + auto-ingest mine) within single Claude Code session #1581 and the broader concurrent-writer cluster it links.
• Root cause and a workaround (pin chroma-hnswlib==0.7.6, set hnsw:num_threads=1): discussion HNSW silent corruption on chromadb 1.5.x — root cause, symptoms, diagnosis, and fix #1686.
• Legacy repair re-mine wording (your point 2): repair (legacy) and MCP server cannot recover from a ChromaDB compactor failure, though --mode from-sqlite can #1843 / fix(repair): point index-read failures to repair --mode from-sqlite (#1843) #1847.
• Lock-file leak (your point 4): already fixed, mine_lock() leaves stale lock files in ~/.mempalace/locks/ #1800 is closed.

[KeilerHirsch]

Follow-up — reproduced again on a 268k-drawer Windows palace, with a concrete trigger and a working prevention for the hardening list:

Trigger (deterministic): a SessionStart hook that kicks an ingest mine overlapping the interactive MCP startup reliably induces the wedge. VSCode restart → Start-ScheduledTask <ingest> → spawns mine → overlaps MCP init → mine aborts mid-write → compactor wedged. So it's not just "interrupted mine" in the abstract — a session-start ingest hook overlapping the MCP is a repeatable way to hit it.

Prevention that worked: guard the ingest trigger so it only spawns a mine when none is already running (process check for mempalace … mine). Eliminated recurrence in our setup.

Repair data point: repair --mode from-sqlite into a fresh path rebuilt 269,741 rows (268,233 drawers + 1,508 closets, 1.78 GB) in ~70 min (~25 MiB/min); MCP status returned instantly after the swap. Verify before swapping: open the chroma client directly on the rebuilt palace — it must return count() without the backfill error.

Secondary Windows symptom: the first MCP start after the swap can fail with Subprocess initialization did not complete within 60000ms because the SessionStart fts5_heal rebuild locks the DB while the MCP server initializes. Self-heals once FTS5 is rebuilt; decoupling fts5_heal from MCP init would remove it.

Env: mempalace 3.4.0, chromadb 1.5.9, Windows 11. Happy to test the chroma-hnswlib==0.7.6 + hnsw:num_threads=1 mitigation from @mvalentsev's comment and report back.

[mvalentsev]

Confirmed the secondary symptom, and it's a separate mechanism from the compactor hang: plain SQLite VACUUM lock contention.

mempalace repair ends in _vacuum_and_rebuild_fts5 (FTS5 rebuild + VACUUM), and VACUUM takes an exclusive lock on chroma.sqlite3 for the whole file rewrite. On a 1.4 GB palace that lock is held for over a minute, past the 60s init budget. The eager-warmup open the server does at startup, before it answers initialize, blocks for the entire VACUUM, which is your did not complete within 60000ms.

This is your hardening point 1. The one thing to flag is that the lock holder here is repair itself, not the compactor or an interrupted mine, so the deadline has to wrap the startup open too.