Runaway HNSW index explosion (582 GB link_lists.bin) on add_drawer to new room in large wing

[marcel10100]

MemPalace v3.3.2 — runaway HNSW write loop created 582 GB link_lists.bin, filled disk to 193 MB free

Summary

A single mempalace_add_drawer call via the MCP server into an existing wing (wing had ~101,626 drawers already) but into a room name that did not exist yet triggered a runaway HNSW index rebuild. Over approximately 20 minutes, ChromaDB wrote a single file link_lists.bin to 582 GB inside the affected collection's segment directory, taking a 2.0 GB palace to ~305 GB total and the user's macOS disk from ~75 GB free down to 193 MB free — effectively hanging the system.

After manually deleting the runaway file (rm link_lists.bin), disk pressure recovered. But mempalace search now fails with Error deserializing pickle file: Operation timed out (os error 60) / Error constructing hnsw segment reader. chroma.sqlite3 (1.9 GB) appears intact.

Environment

• MemPalace: 3.3.2 (installed from git+https://github.com/MemPalace/mempalace.git@v3.3.2)
• Install: pipx install --python python3.13 git+https://github.com/MemPalace/mempalace.git@v3.3.2
• Python: 3.13.12 (pipx venv; system Python 3.14.4 fails pipx due to ensurepip)
• OS: macOS 26.2 (Tahoe), Apple Silicon, 926 GB SSD
• ChromaDB: whatever v3.3.2 pulls (not pinned)
• Palace location: ~/Documents/mempalace/palace/ (custom — edited ~/.mempalace/config.json → palace_path)
• Embedding: default all-MiniLM-L6-v2 (384-dim)

Palace state when blow-up happened

• Total drawers: 171,536 across 18 wings
• Largest wing project01: 101,626 drawers (mined via mempalace mine <dir> --mode convos --wing project01)
• Palace footprint before add_drawer: 2.0 GB
• Palace footprint after: ~305 GB (link_lists.bin = 582 GB per ls -lh, subdir = 304 GB per du)

Trigger (exact MCP call)

```json
{
"tool": "mempalace_add_drawer",
"args": {
"wing": "project01",
"room": "mempalace-setup",
"content": "<~3 KB markdown documenting an install>",
"added_by": "claude-opus-4-7",
"source_file": "session-2026-04-21-mempalace-install"
}
}
```

wing=project01 already had ~101,626 drawers. room=mempalace-setup did NOT exist. Call returned success with a valid drawer_id within ~1 s.

A mempalace_diary_write call to wing_claude fired around the same time and also returned success. That wing is small so unlikely to be the trigger.

Timeline

Time	Event
~17:30	Final mempalace mine completes. 18 wings, 171k drawers, palace 2.0 GB
~17:33	knowledge_graph.sqlite3 last written (Stop hook from prior session)
~17:37	mempalace_add_drawer MCP call → success in ~1 s
~17:37	mempalace_diary_write MCP call → success
~17:56	User notices link_lists.bin = ~600 GB. mempalace_list_wings → MCP error -32000: Connection closed. CLI mempalace search fails with HNSW deserialize timeout.

~19 min between success response and the runaway being caught.

Palace structure when we caught it

```
~/Documents/mempalace/palace/
├── chroma.sqlite3 (1.9 GB)
├── knowledge_graph.sqlite3 (4 KB)
├── 4cd3d9c7-4ac4-4ae1-b97b-3d291698b6b6/ (172 KB — OK)
│ └── link_lists.bin (0 B)
└── 5eed4139-94d5-4e1b-90fb-c0d317745d95/ (304 GB per du)
├── length.bin (400 B)
├── data_level0.bin (167 KB)
└── link_lists.bin (582 GB per ls -lh) ⚠️
```

Default all-MiniLM-L6-v2 produces 384-dim vectors. 101k drawers = ~150 MB of raw vectors. 582 GB is clearly not data — it's HNSW graph explosion.

Symptoms after rm link_lists.bin

• mempalace search "..." → Error deserializing pickle file: Operation timed out (os error 60) / Error constructing hnsw segment reader
• mempalace status → chromadb.errors.InternalError ... too many SQL variables (different error, possibly unrelated collection-size issue)
• MCP server list_wings/search → MCP error -32000: Connection closed until restart

Hypothesized reproduction (NOT yet confirmed)

1. Build a palace with one large wing (~100k+ drawers, many existing rooms)
2. Via MCP, call mempalace_add_drawer targeting that wing with a room name that does NOT exist yet
3. HNSW index rebuild enters a write loop filling link_lists.bin unboundedly

Questions for maintainers

1. Known upper bound on drawers-per-wing before HNSW gets unstable? If yes, please document / enforce in mine.
2. Does writing to a brand-new room in an existing large wing trigger a segment rewrite? That's the suspected bug path.
3. Can chroma.sqlite3 data be recovered via mempalace repair? Would preserve all 171k drawers.
4. Add a disk-usage circuit breaker during writes (abort if file grows >10× expected within 60 s, or free disk drops below threshold)?
5. Should mempalace mine --mode convos auto-cap or warn near known-problematic drawer counts?

Recovery path about to try

• All three MemPalace hooks (SessionStart, Stop, PreCompact) disabled in ~/.claude/settings.json
• About to run mempalace repair offline with disk monitoring

Happy to share sanitized metadata dumps from chroma.sqlite3 or run reproduction on a scratch palace.

[marcel10100]

Follow-up from the reporter — workaround + scale data

We ended up building a sqlite-vec backend as a workaround for this bug, and it's now stable enough that it may be useful context for whoever tackles the root cause (or for anyone else hitting this at scale).

What we built

A new backend at mempalace/backends/sqlite_vec.py (~880 lines) implementing the BaseBackend / BaseCollection ABC using the sqlite-vec extension. Key architectural difference from ChromaDB:

• No HNSW graph. Brute-force cosine similarity scan over the vector table.
• One SQLite file per collection. No segment directories, no link_lists.bin.
• No index rebuild on new metadata filter values — writing to a brand-new room in an existing large wing is just an INSERT, not a graph restructure.

This directly sidesteps the trigger pattern described in the original report.

Proven scale

Running this backend for the past several days on the same workload that triggered the 582 GB blow-up:

Metric	Value
Drawers mined clean	419,767 across 24 wings
Palace size	1.1 GB (vs. 305 GB in the HNSW failure mode)
Query latency (brute-force cosine)	200–600 ms on M-series Apple Silicon
Corruption events under concurrent mine + MCP	0
Initial reproduction of #1091 → post-migration	74k → 419k drawers, no runaway

Vector math at this scale: 419k × 384 dims × 4 bytes = ~644 MB raw. Brute-force is tractable.

Tradeoffs (honest)

• Query latency is linear in drawer count. 200–600 ms at 170k, will grow to ~1 s at 500k. HNSW would stay sub-100 ms if it worked.
• No approximate-NN tuning. You get exact top-k or nothing.
• Extra dependency (sqlite-vec) — but it's a single SQLite loadable extension, no C++ build chain.

For a personal memory system where queries are occasional and correctness matters more than raw QPS, this tradeoff has been fine. For larger multi-user deployments it might not be.

Not proposing a backend swap

Reading the direction in #1000 (quarantine_stale_hnsw), #1071 (thread caps), and #1072 (MEMPALACE_BACKEND env var), it's clear the project is defending ChromaDB as the default, not replacing it. That's a reasonable call — most users won't hit this bug.

But once #1072 lands, a sqlite-vec backend could live alongside ChromaDB as an optional choice for users who keep hitting #1091 at scale. Happy to open a PR along those lines if there's interest. If not, we'll keep it in our fork.

Answers to the original questions (from our experience)

1. Known upper bound on drawers-per-wing before HNSW gets unstable?

We hit it at ~101k drawers in one wing + add to new room. Smaller wings with many rooms were fine.

2. Does writing to a brand-new room in an existing large wing trigger a segment rewrite?

That's consistent with what we observed. The runaway started on an add_drawer with a new room value and compounded over ~19 min of apparent success before the file growth became visible.

3. Can chroma.sqlite3 data be recovered via mempalace repair?

In our case, no — mempalace repair timed out on the HNSW segment reader. We extracted drawer content directly from chroma.sqlite3 via SQL and re-mined into the new backend. ~97% preserved.

4. Disk-usage circuit breaker during writes?

Strongly second this. A simple "abort if any segment file grows >10× the total drawer count in 60 s" would have caught this before it ate 300 GB.

5. Cap or warn near known-problematic drawer counts?

If a threshold exists (e.g. ~80k per wing), a mempalace mine warning would save a lot of pain.

Happy to share the backend code, the migration script, or reproduction steps on a scratch palace. Let us know what's useful.

[1990eam]

Just experienced this. link_lists.bin grew over 100gb in the course of one week. Don't have much more to add, but I hope it gets solved soon 🙏🏻 Reading through this and related issues to see what alternatives I have.