Add MslDbBuilder: FASTA→.msl spectral-library builder for ManySearchTask

[trishorts]

What this PR adds

MslDbBuilder — a standalone .NET 8 console tool that turns protein FASTA databases into mzLib .msl spectral libraries. It is the offline "library compiler" for MetaMorpheus's ManySearchTask: instead of digesting thousands of FASTAs at search time, the search reads pre-built .msl files (precursor mass + charge + Chronologer iRT + sequence, optionally with stored fragments).

It is added as a new project under mzLib/MslDbBuilder/ referencing UsefulProteomicsDatabases and Readers, and it uses the .msl writer/reader API introduced in PR #1036. This PR is therefore stacked on #1036 (mzlib_speclib) — until that merges, the diff here also shows the .msl format work; it collapses to just the builder once #1036 lands. Please review/merge #1036 first.

Plain-language explanation (what it does and why)

A spectral-library search needs a peptide source: the list of candidate peptides (with their fragment masses) to compare against each spectrum. Normally MetaMorpheus generates that on the fly by digesting a FASTA every run. For ManySearchTask we want to search one sample against thousands of organisms at once (e.g. all bacteria), and re-digesting tens of thousands of FASTAs on every search is wasteful and slow.

MslDbBuilder does that digestion once, offline, and writes the result as compact indexed .msl files:

• Reads each FASTA and digests it (trypsin, 0 missed cleavages, length ≥ 7, variable init-Met, Carbamidomethyl on C / variable Oxidation on M, max 2 mods), generating target and reversed-decoy peptides.
• Predicts each peptide's retention time once with Chronologer (an iRT model) and stores it in the index, so the search can use an RT pre-filter.
• Writes a lean library by default: precursor mass + charge + iRT + sequence, no stored fragments (the search re-fragments the handful of candidates that survive the precursor+RT filter — far cheaper than storing fragments for billions of peptides).
• Can emit one .msl per organism, merge many organisms into a single indexed file (each entry tagged db|accession so the search can still report per-organism hits), or build directly into size-balanced shards for production scale.

The payoff at search time: thousands of "databases" load as a few big indexed files instead of thousands of FASTA opens, and most candidates are eliminated by a learned precursor-mass + RT gate before any fragmentation happens.

Modes

• MslDbBuilder [--verify] [--lean] <outDir> <fasta...> — FASTA → one .msl per database.
• --shardbuild <outDir> <numShards> <fasta...|@listfile> — build N lean, size-balanced merged shards directly from FASTAs, streaming, each entry tagged db|accession, with Chronologer iRT (the production path).
• --merge <out.msl> <msl...> — combine many .msl into one merged file.
• --read / --probe / --checkrt / --rtcalib / --streamtest — round-trip, inspection, RT-calibration and writer-test utilities.

How we arrived at the production design (160 shards) — the sizing journey

The goal was a single searchable library covering all ~17,300 bacterial proteomes (~30 GB of input FASTA). Getting there was not obvious; here is everything we tried and why we ended where we did.

1. "Just build one giant merged .msl." — Impossible.
The .msl format stores its entry/precursor count as a 32-bit int (cap ≈ 2.147 B, i.e. 2³¹). The full bacterial set is billions of target+decoy peptide entries — well past that cap. A single merged file simply cannot represent it. → We must shard.

2. "A handful of big shards (~4–6)." — The right instinct, wrong first cut.
Sizing said ~108 bytes/entry, ~11–12× the FASTA size, ~338–377 GB total. A few shards each under the 2.1 B cap looked sufficient on paper, so the first prototype tried a small shard count.

3. First prototype: 150 databases → 3 shards. — 2 of 3 shards crashed.
Two distinct problems surfaced:

• Chronologer file contention: each shard built its own Chronologer predictor in parallel, and they all read the same TorchSharp weights temp file → IOException (file in use). Per-shard parallel predictor construction is not safe.
• Memory blow-up: peak 71 GB for just 150 databases across 3 parallel shards. That does not scale to 17,300 (it would need ~800 GB+).

4. The real wall — the global iRT cache crash (the saga).
iRT is a pure function of the peptide sequence, not the organism, so the same sequence appears in thousands of proteomes. The fix for both speed and the contention above was a global iRT cache: predict each unique sequence once, reuse everywhere. We built it as a single ConcurrentDictionary<string,double?>. It corrupted — throwing NullReferenceException from inside its own TryAddInternal/TryGetValue — once it grew past ~900 M entries, even with hundreds of GB of RAM free. We misdiagnosed it twice: first as a TorchSharp/native crash, then as memory pressure. The true cause is a .NET internal scale limit: a single dictionary's backing bucket array can't resize cleanly at that magnitude. Single-threaded inserts crashed too, so it was not a concurrency misuse.

• Fix: shard the cache into 128 hash-routed sub-dictionaries (ShardedSeqCache, key routed by hash & 127). Each sub-dictionary holds only tens of millions, so every resize is a small allocation. This held 2.30 B unique sequences without issue, and made parallel preload safe (cache reload dropped from ~68 min to ~7 min).

5. Architecture pivot off the back of that.

• Serial prediction with one shared, locked Chronologer — one predictor already saturates every core; N predictors only oversubscribe and re-introduce the temp-file contention.
• Global iRT cache persisted to irt_cache.tsv (kept after the build — it makes re-builds and library extensions nearly free, since the expensive iRT is never re-paid).
• Wave-based shard finalization with checkpointing — shards finalize in waves; each completed .msl is a durable checkpoint, and shard concurrency is decoupled from shard count (MSLDB_SHARD_CONCURRENCY), bounding peak RAM. An interrupt loses at most the in-flight wave's digest+write (iRT is already banked).
• Direct lean-entry build — write the lean POCO directly instead of constructing a LibrarySpectrum/MzSpectrum for fragments that a lean entry doesn't carry (byte-identical output, far fewer allocations).
• Also fixed a real mzLib data bug found here: peptides with an ambiguous residue (X → NaN m/z) poisoned the index range query and returned empty for the whole library; those peptides are now skipped at build time.

6. Choosing the final shard count: 160.
Three constraints drove it:

• (a) Each shard must stay under the 2³¹ entry cap.
• (b) Keep every database whole in exactly one shard. This is the key design choice: if no database is split across shards, the per-database union across shards is clean and the search never has to merge partial results — it just searches each shard and attributes hits per db|accession.
• (c) Balance load and bound per-shard RAM. Smaller shards mean a smaller working set per shard load and smoother parallelism.

Greedy size-balanced packing of 17,300 whole databases into shards of ~2 GB lands at 160 shards: ~18 M entries and ~2.25 GB per shard, ~108 databases per shard, 358 GB total. (Two shards would technically fit under the cap, but each would be ~180 GB to load — unworkable; 160 keeps each shard cheap to load and resumable.)

7. Final build & search numbers.

• Build: 160 shards in 7.2 h, 0 restarts, peak 473 GB (server-GC-capped, no OOM), 2.30 B unique Chronologer iRT predictions cached (irt_cache.tsv, 127 GB, kept).
• Search (160 shards × 3 raw files): large growing heaps need server GC — under workstation GC the per-shard load degraded from ~22 s to ~106 s; server GC keeps it flat. Full search ran in ~1.9 h, peak 325 GB (server-GC-capped, no OOM), human base 1,769 PSMs @ 1% FDR, all 17,300 organisms summarized.

Root-cause lessons (carried forward)

• A single ConcurrentDictionary corrupts past ~900 M entries regardless of free RAM — shard large dictionaries (this cost a multi-day misdiagnosis: TorchSharp → memory pressure → the real .NET scale limit).
• Large, growing heaps need server GC for both build and search; workstation GC causes serial full-GC stalls at this scale.

Notes for reviewers

• This is a console tool, added under mzLib/MslDbBuilder/. If the team would prefer it live elsewhere (e.g. a tools/ folder or a separate repo), say the word.
• It depends on the .msl API from MSL mzLib Spectrum Library indexed binary format #1036, so its base is mzlib_speclib; it will not build against master until MSL mzLib Spectrum Library indexed binary format #1036 merges.

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 93.17872% with 200 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.33%. Comparing base (92be818) to head (ddf7b22).
⚠️ Report is 3 commits behind head on master.

Files with missing lines	Patch %	Lines
mzLib/Readers/SpectralLibrary/MslConverter.cs	0.00%	44 Missing ⚠️
mzLib/Readers/SpectralLibrary/MslWriter.cs	95.41%	15 Missing and 26 partials ⚠️
mzLib/Omics/SpectralLibrary/MslFragmentIon.cs	89.23%	13 Missing and 8 partials ⚠️
mzLib/Readers/SpectralLibrary/MslReader.cs	95.37%	6 Missing and 14 partials ⚠️
mzLib/Omics/SpectralMatch/Mslindex.cs	95.20%	3 Missing and 10 partials ⚠️
...zLib/Readers/SpectralLibrary/MslProteoformIndex.cs	91.93%	3 Missing and 7 partials ⚠️
...n/Chronologer/ChronologerRetentionTimePredictor.cs	85.48%	4 Missing and 5 partials ⚠️
mzLib/Readers/SpectralLibrary/MslLibrary.cs	97.42%	4 Missing and 5 partials ⚠️
...Lib/Readers/SpectralLibrary/MslProteoformScorer.cs	93.43%	5 Missing and 4 partials ⚠️
mzLib/Readers/SpectralLibrary/MslMerger.cs	95.93%	2 Missing and 5 partials ⚠️
... and 4 more

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1076      +/-   ##
==========================================
+ Coverage   81.63%   82.33%   +0.69%     
==========================================
  Files         369      382      +13     
  Lines       47605    50509    +2904     
  Branches     5649     6102     +453     
==========================================
+ Hits        38863    41586    +2723     
- Misses       7643     7734      +91     
- Partials     1099     1189      +90     

Files with missing lines	Coverage Δ
mzLib/Omics/SpectralLibrary/MslSpectralLibrary.cs	100.00% <100.00%> (ø)
...SpectralMatch/IHasSequenceCoverageFromFragments.cs	100.00% <ø> (ø)
...zLib/Readers/SpectralLibrary/Mslfiletypehandler.cs	100.00% <100.00%> (ø)
...ts/Koina/AbstractClasses/FragmentIntensityModel.cs	89.66% <89.65%> (+0.39%)	⬆️
mzLib/Readers/SpectralLibrary/SpectralLibrary.cs	89.03% <96.47%> (+3.65%)	⬆️
mzLib/Omics/SpectralLibrary/MslStructs.cs	66.66% <66.66%> (ø)
mzLib/Readers/SpectralLibrary/MslLibraryData.cs	89.28% <89.28%> (ø)
mzLib/Readers/SpectralLibrary/MslMerger.cs	95.93% <95.93%> (ø)
...n/Chronologer/ChronologerRetentionTimePredictor.cs	83.58% <85.48%> (+0.75%)	⬆️
mzLib/Readers/SpectralLibrary/MslLibrary.cs	97.42% <97.42%> (ø)
... and 7 more

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.