PreCompact hook: double-ingest race condition causes HNSW corruption (no palace-wide write lock)

[apollion69]

Summary

MemPalace 3.3.3 has a race condition in hooks_cli.py that causes concurrent ChromaDB writes when the PreCompact hook fires, leading to HNSW index corruption.

Root cause

hook_precompact() at line ~656 calls two ingest paths without any palace-wide lock:

1. _ingest_transcript(transcript_path) — spawns an async subprocess.Popen (fire-and-forget)
2. _mine_sync(...) — runs a sync subprocess.run immediately after

Both paths write to the same ChromaDB collection. If a Stop/SessionEnd hook or a background mempalace mine is already running, the result is two concurrent HNSW writers.

Neither path is gated by the existing mine.pid PID guard — _ingest_transcript bypasses it entirely, and _mine_sync is a separate code path.

Observable symptom

hook.log shows:

chromadb.errors.InternalError: Error in compaction: Failed to apply logs to the hnsw segment writer

In our case, link_lists.bin (HNSW higher-level connections) grew from ~50 MB to 210 GB apparent / 41 GB real (sparse file expansion) before we caught it.

Reproduction

1. Enable both PreCompact and SessionEnd hooks in Claude Code settings.json
2. Have a long session (so both hooks fire close together on context compaction)
3. Observe HNSW errors in hook.log; du will show the palace growing

Proposed fix

Add a palace-wide fcntl.flock(LOCK_EX) in hooks_cli.py before any HNSW write operation. The lock file should be shared across all three write paths:

• hook_stop() → _ingest_transcript() / _maybe_auto_ingest()
• hook_precompact() → both ingest calls
• CLI mempalace mine command

Example pattern (already working in our workaround):

import fcntl, os
LOCK_FILE = os.path.join(palace_dir, ".palace-write.lock")
with open(LOCK_FILE, "w") as lf:
    fcntl.flock(lf, fcntl.LOCK_EX)
    # ... all HNSW writes here

Workaround (used in production)

We disabled PreCompact entirely (no-op exit 0 script) and moved conversation mining to a cron job with a shared flock -n guard. See the monkey-patch approach in stop_diary_only.py that disables the async paths while keeping the diary checkpoint.

Environment

• MemPalace 3.3.3
• ChromaDB (bundled version)
• Claude Code SessionEnd + PreCompact hooks enabled
• Host: WSL2 on Windows

Related

• Single-slot mine.pid PID guard has no O_EXCL and is bypassed by both _ingest_transcript (Popen) and _mine_sync paths — worth hardening separately

[mvalentsev]

The race you describe is real. PR #1162 (imtylervo, OPEN since 2026-04-24) closes it architecturally cleaner: it acquires mine_palace_lock inside ChromaCollection.add/upsert/update/delete directly, with a re-entrant guard so calls from inside miner.mine() don't self-deadlock. Effect for your scenario:

• _ingest_transcript Popen -> mempalace mine -> mine_palace_lock (pre-existing from fix: HNSW graph corruption, PreCompact deadlock, mine fan-out (closes #974, #965, #955) #976)
• _mine_sync -> mempalace mine -> mine_palace_lock (pre-existing from fix: HNSW graph corruption, PreCompact deadlock, mine fan-out (closes #974, #965, #955) #976)
• MCP tool_add_drawer / any direct-backend writer -> ChromaCollection write -> mine_palace_lock (new in fix: serialize ChromaCollection writes through palace lock #1162)

All three paths serialize through the same flock once #1162 lands. Locking at the hooks_cli layer (your proposal) closes the hook->hook race but leaves the hook->MCP race open; #1162's seam is broader.

Cluster of open issues with the same root cause: #1092 (umbrella, hooks + MCP + CLI), #1083 (Stop + PreCompact transcript-parent mining), #1212 (Stop hook concurrent mempalace mine), #357 (original parallel mining HNSW corruption).

[igorls]

Following @mvalentsev's 2026-04-28 note: PR #1162 merged to develop on 2026-05-06 and lands the architectural fix — ChromaCollection.add/upsert/update/delete now acquire mine_palace_lock(palace_path) with a re-entrant guard, so the PreCompact _ingest_transcript Popen and the immediately-following _mine_sync cannot enter Chroma's HNSW write path concurrently anymore.

If you have an environment where the original repro (PreCompact + SessionEnd both wired up, long session) is reproducible, a re-run against develop tip would let us close this in v3.3.5.

[apollion69]

Verification result: confirmed fixed on develop tip.

Installed develop (post-#1162) in an isolated venv and ran both lock test suites from the PR:

```
platform linux -- Python 3.12.3, pytest-9.0.3
collected 12 items

test_chroma_collection_lock.py::test_palace_path_none_skips_lock PASSED
test_chroma_collection_lock.py::test_writer_blocks_during_mine PASSED
test_chroma_collection_lock.py::test_reentrant_inside_mine_passes_through PASSED
test_chroma_collection_lock.py::test_concurrent_writers_serialize PASSED
test_chroma_collection_lock.py::test_read_path_does_not_acquire_lock PASSED
test_palace_locks.py::test_single_acquire_succeeds PASSED
test_palace_locks.py::test_lock_reusable_after_release PASSED
test_palace_locks.py::test_same_palace_serializes_across_processes PASSED
test_palace_locks.py::test_different_palaces_dont_conflict PASSED
test_palace_locks.py::test_palace_path_is_normalized PASSED
test_palace_locks.py::test_reentrant_same_thread_passes_through PASSED
test_palace_locks.py::test_mine_global_lock_is_alias_for_back_compat PASSED

12 passed in 1.67s
```

ChromaCollection.add/upsert now go through mine_palace_lock with the re-entrant guard — both the hook→mine and the hook→hook races are closed. Environment: WSL2 / Python 3.12.3, same setup where the original HNSW corruption (link_lists.bin sparse-expansion) was observed.

Good to close for v3.3.5.