Bound FTS maintenance across imports and projection rebuilds

[j2h4u]

Problem

Initial history ingestion could make SQLite's WAL and FTS maintenance disproportionately expensive. In the real case that prompted this work, importing a roughly 260 MB Hermes database grew the WAL to about 9–10 GB, and an uninterrupted run could keep growing. The same risk was present in other provider importers and projection rebuilds because their write/maintenance lifecycles were not shared.

This is especially painful on small VPS installations: the source data and final CTX database fit comfortably, but transient FTS work can exhaust the disk before the import completes.

What changed

• introduce one bounded bulk-write lifecycle shared by provider imports, archive ingestion, and search-projection rebuilds;
• rotate transactions by both serialized bytes and unit count, checkpoint between bounded rotations, and cap deferred maintenance under pinned readers;
• distinguish recoverable lock/checkpoint pressure from fatal errors such as SQLITE_FULL and corruption;
• persist bulk-mode and projection-rebuild state so interrupted work resumes idempotently on the next writable open;
• keep record and event projection replacement idempotent when ordinary writers interleave with bounded rebuilds;
• hold the projection marker check and every lexical/semantic projection query in one SQLite snapshot, returning an explicit error instead of partial results during an incomplete rebuild;
• preserve the authoritative first Codex session_meta and the actual source file identity during batched imports.

The implementation is provider-agnostic. Importers use the same BoundedBulkWriteTransaction rather than carrying provider-specific checkpoint loops.

Validation

• clean real-host import across six discovered runtimes:
  • about 4.5 GB / 10.2k source files;
  • about 551k events / 5.6k sessions;
  • completed in 22m51s with an observed WAL peak around 7.5 MB;
• repeat import completed idempotently with only live-session additions;
• SQLite PRAGMA quick_check: ok;
• ctx-history-store: 114 tests passed;
• ctx-history-capture: 261 passed, 1 manual performance benchmark ignored;
• complete ctx CLI unit and integration suites passed;
• final release binary deployed against the real database: 552,296 indexed events, 5,675 sessions, zero pending or failed source-import files, lexical search successful.

I also tested larger batch profiles rather than assuming that larger transactions were better. 32 MiB and 64 MiB profiles reproduced SQLITE_FULL; 16 MiB triggered an unwanted full FTS segment merge in the Hermes regression test. The submitted code therefore keeps the validated 8 MiB / 64-unit bound.

The final diff received independent adversarial reviews from DeepSeek V4 Pro and GLM 5.2. Findings around incomplete semantic reads, projection TOCTOU, and concurrent record-projection duplicates were fixed and re-reviewed before opening this PR.

[j2h4u]

Fresh validation against current upstream/main clarified the direction here.

On a real multi-provider dataset (4,361 files / 3.9 GiB across all six discovered sources), a full foreground import completed with the WAL remaining bounded and absent after checkpoint. An immediate second --resume pass also kept the WAL bounded, but still took 9m12s and left the same inventory state pending (202 catalog sessions and 3 source-import files). That is a convergence/publication-state issue rather than the bulk-transaction design in this PR.

The open #172 series now provides the broader architecture we wanted from this PR: bounded and resumable provider publication, fresh-first scheduling, persistent bounded FTS maintenance, disk pacing, and cross-provider incremental refresh tests. Rebasing this PR would preserve a competing implementation rather than add value, so I am closing it as superseded by #172.

One narrower idea here is not necessarily covered by that conclusion: commits d2f2038 and eb1fc59 make an explicit full search-projection rebuild bounded/resumable and reject reads while the projection is incomplete. After #172 lands and is validated on the same dataset, we can reassess that behavior independently and, if still useful, propose it as a small focused change.