Wire `quarantine_stale_hnsw` into backend open path (follow-up to #1000)

[messelink]

#1000 landed the quarantine_stale_hnsw() primitive but, as noted in that PR, intentionally did not wire it into the backend open path:

Additive only. Exposes quarantine_stale_hnsw() as a helper. Intentionally not wired into _client() / startup on this PR — the goal is to land the primitive first so operators and higher layers can opt in. A natural follow-up is to call it automatically on palace open behind an env var or a config flag; happy to do that in a second PR once this shape is agreed.

Filing this as the tracking ticket for that follow-up.

Why it matters today

A palace hit the exact drift failure mode #1000 describes. Symptoms matched #823 / neo-cortex-mcp#2:

• mempalace status — SIGSEGV
• mempalace repair --yes — SIGSEGV (also crashes before it gets anywhere useful)
• MCP server under Claude Code — connection drops 1s into each list_wings / search call, with "Failed to apply logs to the hnsw segment writer" in the stderr log

Three segments were 4–10 days out of sync between chroma.sqlite3 and data_level0.bin. After manually running quarantine_stale_hnsw(palace_path), everything worked immediately: status, search, MCP.

So the primitive does what it says — the gap is that an affected user has no path to recovery through the shipped CLI. mempalace repair is the natural entry point for "my palace crashes on open," and it currently segfaults before reaching any recovery code because get_collection() loads the HNSW index first.

Proposal

Two places it would help:

1. cmd_repair (highest-leverage) — call quarantine_stale_hnsw() before the first get_collection() / count() call. A repair command that can't be run on a corrupted palace is a repair command that doesn't do its job.

2. Backend _client() open path, behind a config flag — opt-in via hnsw.auto_quarantine_drifted = true in ~/.mempalace/config.json or a MEMPAL_AUTO_QUARANTINE=1 env var. The 1-hour default threshold is conservative enough that this should be safe to enable for most users who've been bitten. Keep it opt-in rather than default so the heuristic doesn't quietly quarantine segments on someone else's setup.

Related

• feat(backends): quarantine_stale_hnsw — recover from HNSW/sqlite drift (closes #823) #1000 — primitive landed
• ChromaDB: HNSW persistence vs add_drawer — default sync_threshold can leave semantic search stale #823 — original drift / stale-results report
• fix: HNSW graph corruption, PreCompact deadlock, mine fan-out (closes #974, #965, #955) #976 — sibling INSERT-path race (different failure mode)
• fix: degrade gracefully when filtered Chroma search fails #951 — graceful fallback for filtered query failures (different failure mode)

[tvlgiao]

Empirical confirmation — same drift, same fix, on Claude Code Stop hook

Hit this on mempalace 3.3.3 + chromadb 1.5.8 + Apple bundled Python 3.9.6 (arm64). Stop hook (mempal-stop-hook.sh) was segfaulting every save:

Stop hook error: Failed with non-blocking status code: ...
50832 Segmentation fault: 11  mempalace hook run "$@"

Faulthandler trace pinpointed chromadb/api/rust.py:484 in _add on the Stop-hook diary write path:

chromadb/api/rust.py:484 _add  ← SEGV
chromadb/api/models/Collection.py:118 add
mempalace/backends/chroma.py:195 add
mempalace/mcp_server.py:965 tool_diary_write
mempalace/hooks_cli.py:411 _save_diary_direct
mempalace/hooks_cli.py:591 hook_stop

chroma.sqlite3 was 5212 s newer than data_level0.bin on both segment dirs — exact drift profile #1000 / #823 describe. count() also segfaulted — no recovery surface from the CLI side, matching what you observed with repair.

Local patch — wire into _client() open path

Two-line patch to backends/chroma.py:

         if cached is None or inode_changed or mtime_changed or mtime_appeared:
+            quarantine_stale_hnsw(palace_path)
             _fix_blob_seq_ids(palace_path)
             cached = chromadb.PersistentClient(path=palace_path)

     @staticmethod
     def make_client(palace_path: str):
         ...
+        quarantine_stale_hnsw(palace_path)
         _fix_blob_seq_ids(palace_path)
         return chromadb.PersistentClient(path=palace_path)

After patch, on next Stop hook fire:

Quarantined stale HNSW segment .../2bf7bf7c-... (sqlite 5212s newer than HNSW); renamed to ....drift-20260425-145716
Quarantined stale HNSW segment .../731ec6bb-... (sqlite 5213s newer than HNSW); renamed to ....drift-20260425-145716
Diary entry: diary_wing_search_20260425_145716603786_a29ca2bc3989 → wing_search/diary/checkpoint
{"systemMessage": "✦ 2 memories woven into the palace — stop, hook, error"}

26535 drawers preserved. Old segments quarantined (recoverable), new segment rebuilt lazily by chromadb.

On opt-in vs default

I picked unconditional wiring locally because the failure mode is silent (Stop hook is non-blocking — Claude Code logs and continues, user sees no error, save just disappears). Opt-in via env/config means most affected users will never know to opt in.

That said, I take your point about not quietly quarantining segments on someone else's setup. Middle ground worth considering:

• cmd_repair: unconditional, as you proposed — agree this is highest-leverage.
• _client() open path: default ON when chroma.sqlite3 mtime − HNSW mtime > threshold AND the process is about to write (not on read-only opens). Keeps reads safe, auto-recovers writes that would otherwise SEGV. Still surface a one-line warning to stderr so it's not invisible.

Happy to send a PR if either shape is wanted — name your preference and I'll target develop.

Repro environment

• macOS 25.2.0, arm64 (M-series)
• Python 3.9.6 (Apple Command Line Tools bundled)
• mempalace 3.3.3, chromadb 1.5.8, numpy 2.0.2, torch 2.6.0
• Harness: Claude Code plugin via mempal-stop-hook.sh

[messelink]

Thanks for the independent repro on a totally different stack — Apple-bundled Python 3.9 on arm64 is about as far from our Linux + pipx setup as you can get, and the same drift profile (chroma.sqlite3 newer than data_level0.bin by hours, count() segfault, repair segfault) puts to bed any "weird local config" theory.

Your argument about the silent failure mode is the right correction to our opt-in framing in this issue. Stop hook non-blocking + save just disappears + no user-visible error = users never know to opt in. The only people who'd flip the env var are people who already know the bug exists, which is the wrong cohort to be defending against the bug.

The compromise shape reads right to us:

• cmd_repair — unconditional. Already documented as the "my palace is broken" entry point; it should actually work on a broken palace.
• _client() open path — auto-quarantine when (a) mtime gap exceeds threshold and (b) the process is going to write. Read-only opens stay strictly observational. Stderr warning so the recovery is at least visible in logs even if the user doesn't notice in the UI.

The write-vs-read gating is the bit that addresses the "don't quietly quarantine on someone else's setup" concern from PR #1000's wording. Read paths can still surface stale results (the original #823 symptom), but they won't suddenly rearrange disk on the user's behalf.

If you send the PR, we'll review the diff against this shape and test it locally against our (currently quarantined) palace.

[messelink]

#1173 shipped this on 2026-04-26 — quarantine_stale_hnsw() is now called inside make_client() so every fresh PersistentClient open is protected (CLI, hooks, MCP server, tests). The 1h → 5min threshold change also addresses @tvlgiao's "0.96h drift caused segfault" point.