Status: authoritative. Supersedes earlier baseline and RLS result docs. Audit duration: ~6 hours across two machines (RTX 3070 Ti + WSL2 / Apple Silicon).
This document is the forensic record of a bench audit that started from "RLS isn't helping, recall is 4%" and ended with two distinct dataset bugs, one retracted baseline, and one bench-adapter feature stack that turned out to be net-positive once measured against honest data.
- The dataset preparation script
benchmarks/scripts/prepare_phase1_datasets.pyhad a two-stage bug that corrupted every LoCoMo benchmark run since it was first committed (2026-04-22, commitd9ecc1c). - The "68.2% LoCoMo baseline" measured the lexical fallback adapter, not the native KV engine. On the Apple Silicon machine that ran the historical numbers, no CUDA was available, so the bench fell back to in-memory word-overlap. That fallback genuinely scored 66.5% R@1 on self-questions; the number is real but it measured a different system than the public narrative implied.
- The native KV engine, measured honestly on a clean dataset, scores ~36% R@1 at 50 LoCoMo items through the production bench adapter, and ~20% R@1 through a minimal probe with no reranker / chunking / quantization tier. The full feature stack is net positive on clean data — opposite of the conclusion the broken dataset implied.
- All four reformulation strategies (RLS) underperform the no-RLS baseline on the clean dataset. Keyword expansion loses 5.3pp; the DeepSeek agent reformulator loses 12.7pp while adding ~1.7s latency per query. The "RLS = 0% improvement" experiments recorded in prior docs were measuring the lexical fallback or a broken dataset; the honest verdict is that RLS as currently implemented hurts retrieval, not "is neutral".
- The bench adapter was scoring a different task than every
LoCoMo leaderboard system.
answer = mapped.ground_truthcollapses LoCoMo's 27.6% duplicate-context items and is not the protocol the LoCoMo paper, Mem0, Memobase, ByteRover, or MemMachine use. The retrieve→LLM-generate→LLM-judge pipeline (Phase 1B.5) now ships with--system tardigrade-llm-gated, validated end-to-end against DeepSeek on the smoke corpus. The full-corpus LoCoMo Judge number is one ~$0.57 + ~1hr GPU job away.
The session began with a CUDA bench run on RTX 3070 Ti showing ~4% deterministic score on LoCoMo, vs a claimed 68.2% baseline. After killing several long-running benches and burning ~3 hours auditing retrieval / hook / quantization code that turned out to be fine, the root cause was found in the dataset prep script.
LoCoMo source items reference evidence by string keys like "D1:3"
(session 1, dialogue turn 3). The prep script accepted only int
keys:
# benchmarks/scripts/prepare_phase1_datasets.py (broken)
for eid in evidence_ids:
if isinstance(eid, int) and eid in dia_to_text:
evidence_lines.append(dia_to_text[eid])Every eid was a string, so the check always failed, evidence_lines
was always empty, and every item silently fell through to
context = full_context. In --locomo-context evidence mode the
output was supposed to be per-item evidence sentences (30-200 chars
each); it was actually the full ~62,000-character conversation for
every item in a 10-conversation, 1542-question corpus.
Effect: 20 items per conversation share one 62K-char context.
For the native KV engine this destroys retrieval — every cell in a
conversation has indistinguishable hidden-state keys. For the lexical
fallback adapter, contexts are also identical, but the question
field is unique per item, and _InMemoryStore.scored_best_match
scores on context + question. The exact question wins by
overlap-with-itself, so lexical recovers 66.5% R@1.
Fix: accept both int and str keys.
if isinstance(eid, (int, str)) and eid in dia_to_text:
evidence_lines.append(dia_to_text[eid])Same fix applied to dia_id accumulation a few lines up.
After Stage 1 was fixed, 1538 of 1542 items had proper per-item
evidence (typically 30-200 chars). The remaining 4 had empty
evidence arrays in the LoCoMo source ("evidence": []). The prep
script still fell through to full_context for those 4 items.
In native-engine mode, items go through TextChunker at ingest.
A 200-char evidence text produces 1 chunk; the 62K-char full
conversation produces 128 chunks. Four items × 128 chunks = 512
"noise" cells from items with no oracle context, competing against
1538 single-cell items in the same corpus. The four noise items
dominated retrieval because their cells covered the entire
conversation's vocabulary.
Concrete measurement: before this fix, the bench adapter scored R@1 = 6% (strict) on 50 LoCoMo items, of which only 3 items had empty-evidence carpet bombing. After the fix, R@1 = 36%.
Fix: in evidence mode, skip items with no evidence rather than falling back to full conversation. They have no oracle context anyway; including them is silently destructive.
if context_mode == "evidence":
if not evidence_lines:
continue # oracle mode: no evidence = no item
context = "\n".join(dict.fromkeys(evidence_lines)).strip()
else:
context = full_contextResulting corpus: 1533 LoCoMo rows (0.6% loss) + unchanged 500 LongMemEval rows.
On the Apple Silicon machine that produced
docs/bench/locomo-longmemeval-baseline.md, the bench harness ran
the TardigradeAdapter in in_memory mode because the machine had
no CUDA. The lexical store's best_match returns the top-scoring
item's ground_truth verbatim. For self-retrieval against the same
1542 items, with unique questions per item, the lexical word-overlap
matcher picks the right item ~66% of the time and returns the
verbatim correct answer, scoring exactly 1.0 on the deterministic
evaluator.
Distribution of scores on the 1542-item historical run:
- 66.5% of items: score = 1.0 (lexical picked correct item)
- 28.3% of items: score = 0.0 (wrong item, zero token overlap)
- 5.1% of items: partial (0.0 < score < 1.0)
The 68.2% number is a real measurement of the lexical
fallback's self-retrieval performance. It is not a measurement of
the native KV engine. The error in the published baseline doc was
framing it as the native engine's performance. The bench output
includes evaluator_mode: "deterministic_fallback" per item, which
should have been read as "the LLM judge never ran" — but the
adapter-mode metadata was not surfaced in the headline.
This also invalidates:
- "Vague-query vocabulary mismatch is the retrieval ceiling" — the
conclusion was drawn from data where every item had identical
context, so "vocabulary mismatch between query and context" was
not a coherent claim. The DeepMind LIMIT paper citation in
docs/refs/external-references.mdas theoretical support for our empirical ceiling is unsupported by our data; the LIMIT paper itself is unaffected, the citation context is wrong. - The 67.2% LLM-judge follow-up
(
docs/bench/locomo-longmemeval-llm-judged.md): also lexical mode, also a different-system measurement.
Two probe paths, both Qwen3-0.6B on CUDA, refinement=centered, query_layer=18, layer ratio 0.67:
| Path | Features | R@1 (50 items) | R@5 |
|---|---|---|---|
| Minimal probe | 1 cell/item, no chunking, no batching, no INT8 tier, no reranker | 20% | 58% |
| Bench adapter | Chunking + GPU batch=8 + INT8 lazy tier + cross-encoder reranker + mem_write_batch | 36% | not measured |
The full bench-adapter stack adds 16pp of R@1 vs the minimal probe. The reranker and feature stack are net positive when the corpus is clean. The historical claim that the bench adapter was "leaking recall" was an artifact of the empty-evidence carpet-bombing — items with 128 chunks each dominate the cell pool and bias the top-1 toward the wrong items.
Same configuration, full clean dataset, single seed (42):
| Dataset | n | avg score | score = 1.0 (R@1 exact) | score = 0.0 (miss) | partial |
|---|---|---|---|---|---|
| LoCoMo | 1533 | 29.62% | 410 (26.7%) | 946 (61.7%) | 177 |
| LongMemEval | 500 | 3.14% | 5 (1.0%) | 439 (87.8%) | 56 |
| Combined | 2033 | 23.11% | — | — | — |
LoCoMo 29.62% (deterministic) is the honest tardigrade-on-Qwen3-0.6B LoCoMo number on the clean evidence-only corpus. Comparable to published baselines (with the heavy caveat that those use different evaluators): Mem0 66.9%, Letta 74%, vanilla GPT-4o-mini 74%, ByteRover 92.2%. The gap to even the vanilla baseline is real and points to either a larger capture model, a trained retrieval head, or a different retrieval signal entirely.
LongMemEval 3.14% is the bigger story. The historical "90.9%" number was the lexical fallback adapter scoring on the question field; the native KV engine, asked to find the one specific evidence chunk among ~40,000 cells (500 items × 25-150 chunks each), almost never wins. Per-token Top5Avg on raw hidden states cannot resolve needle-in-haystack at this model size. Hypotheses for the collapse:
- Chunking dilutes per-item identity (no single chunk contains both the question semantics and the answer text).
- Top-K is dominated by chunks from items with similar surface vocabulary, not the item that contains the actual answer.
- Raw next-token-prediction hidden states aren't trained to be a retrieval signal; the haystack task amplifies that mismatch.
A trained retrieval head, hybrid lexical-plus-latent retrieval, or a larger capture model are the architectural levers worth testing for LongMemEval.
50-item LoCoMo, deterministic evaluator, all bench-adapter features enabled, single ablation per row:
| TDB_RLS_MODE | Avg score | Latency (median) | Delta vs none |
|---|---|---|---|
| none | 21.95% | ~70 ms | — |
| keyword | 16.62% | ~250 ms | -5.3 pp |
| agent (DeepSeek) | 9.29% | ~1700 ms | -12.7 pp |
All RLS modes underperform the no-RLS baseline. The agent
reformulator (DeepSeek deepseek-chat API) is the worst — it adds
~1.7s/query of LLM latency and loses 12.7pp of recall. Hypotheses
worth testing if RLS is to be salvaged:
- Fusion logic discards good answers. The original query likely retrieves the correct item; reformulations add wrong items and the fusion picks one of them.
- Reformulations dilute the latent signal. Replacing "When did Caroline go to LGBTQ support group?" with synonyms changes the Q-side hidden states in ways that don't move closer to the K-side stored representation.
- Confidence threshold is miscalibrated. It was tuned (1.5 → 1.10) against the broken baseline, so the calibration is meaningless.
Recommended action: deprioritize RLS until either the fusion logic is rebuilt or evidence emerges that it helps on a different retrieval architecture (e.g., a trained embedding head rather than raw hidden states).
The bench bug invalidates a narrow set of LoCoMo-specific conclusions, not the broader engineering work.
Still valid:
- 663 tests passing on the core engine (304 Rust + 359 Python). None depend on the LoCoMo data.
- Synthetic-corpus recall: 100% at 100 memories, 100% at 5K memories (Top5Avg, Q4 pipeline). These use synthetic distinct corpora and measure what the engine actually does on well-separated keys.
- KV injection equivalence to text-in-prompt (9/10 recall on synthetic gibberish facts with Qwen3-0.6B). Distinct experimental setup, not affected.
- Cross-model retrieval (same-family 90% R@5 via linear projection, cross-family 76.7% via MLP adapter). Synthetic corpora, unaffected.
- vLLM connector hardening, multi-agent isolation, semantic edges,
multi-view consolidation v2, file ingestion (
TextChunker,FileIngestor,TardigradeClient). Each measured against its own targeted experiment, not LoCoMo. - Cross-encoder reranker, mean-centering refinement, INT8 lazy
tier, GPU batching,
mem_write_batch— each demonstrably improves recall over the minimal probe on the clean dataset (probe 20% → bench 36% at 50 items). - Architectural patterns (Strategy, Bridge, Factory, Active Object, Decorator, Adapter, Repository). Code quality is independent of the dataset.
Retracted or invalidated:
- 68.2% LoCoMo deterministic baseline
(
docs/bench/locomo-longmemeval-baseline.md) — measured the lexical fallback adapter, framed as native engine. - 67.2% LoCoMo LLM-judge baseline
(
docs/bench/locomo-longmemeval-llm-judged.md) — same. - "Vague-query vocabulary mismatch is the retrieval ceiling" — the conclusion was drawn from data where context was identical across items; the claim has no empirical support from these runs.
- All RLS technique-elimination experiments documented in
docs/experiments/(keyword, multiphrasing, embedding, generative 3B, agent) recorded as "0% improvement on LoCoMo". Those numbers were measured against the broken baseline. On the clean dataset, RLS is actively harmful (-5 to -13 pp). - The DeepMind LIMIT paper citation in
docs/refs/external-references.mdas theoretical support for our measured ceiling. The paper is unaffected; the application of it to our empirical results is not justified by data.
Detailed in docs/superpowers/plans/2026-05-14-retrieval-debug-plan.md. This section
records the measured results.
Built scripts/probe_rank_diagnostic.py that, for each query,
records the rank at which the "expected" chunk (proxy: max
ground-truth-token overlap) appears in the latent retriever's top-K.
Bypasses the cross-encoder reranker so we can see the raw retrieval
signal.
LoCoMo (1533 items, top-K=100):
| Rank | Cumulative R@K |
|---|---|
| R@1 | 5.87% |
| R@5 | 19.83% |
| R@10 | 27.40% |
| R@25 | 38.69% |
| R@50 | 50.24% |
| R@100 | 65.56% |
| Outside top-100 | 34.44% |
The cross-encoder reranker lifts deterministic-eval bench score from 5.87% (raw R@1) to 29.62% — a 5× boost by pulling the right chunk from rank 5-100 to rank 1. Reranker does nearly all the recall work; the 34% of items where the right chunk isn't in top-100 are the headroom Phase 1 needs to address.
LongMemEval (500 items, top-K=25 due to compute cost at full top-K):
| Rank | Cumulative R@K |
|---|---|
| R@1 | 0% |
| R@5 | 0% |
| R@10 | 0% |
| R@25 | 0% |
| Outside top-25 | 100% |
Catastrophic — the expected chunk never appears in top-25.
Forensic inspection (scripts/probe_hub_cells.py) showed only 9
unique cells appeared in top-1 across all 500 queries; cell 225
alone was in top-10 of 99.2% of queries. A handful of "hub cells"
dominate all retrievals regardless of query content.
All identified hub cells started with mid-word fragments — typically
mid-number — e.g., "0000 is a great option...", "000 miles...",
"88 as a rough estimate...", "000 PQMs...". Pattern traced to
the chunker producing fragment chunks that carry anomalous hidden
states which score against any query.
TextChunker._split_tokens (python/tardigrade_hooks/chunker.py)
accepted a BoundaryStrategy in __init__ but never called it
inside the splitter. Chunks were sliced purely by token count.
Fix: invoke self._boundary.find_split on every non-final chunk
whose char-space end would otherwise fall mid-word.
Also added ParagraphBoundaryStrategy — prefers \n\n (turn
boundary in conversational transcripts) → sentence end → whitespace.
Wired as default in the bench adapter.
Re-measured diagnostic with end-trim only:
LoCoMo: R@25 38.69% → 54.79% (+16.1pp middle-rank lift).
LongMemEval: still 0% R@25. Hub-cell pattern unchanged in shape — 8 cells in 87-100% of top-10s, just different cell IDs (13940, 24108, 23874, etc.). All still numeric-fragment starts.
The end-only fix changed cell ends but not cell starts. Each non-first chunk's start is determined by overlap-token rollback into the previous chunk — which lands mid-word almost every time for sub-word tokenizers.
Forward-snap chunk start_char to the next whitespace within the
BOUNDARY_LOOKBACK_RATIO-sized window when the preceding character
is alphanumeric. Mirrors the end-trim's lookback logic. Non-zero
overlap no longer produces mid-word starts.
Re-measured diagnostic with both ends and starts trimmed:
| Dataset | Metric | Broken | End-only | Start-trim (Phase 1A.1b) |
|---|---|---|---|---|
| LoCoMo | R@25 | 38.69% | 54.79% | 54.67% |
| LongMemEval | R@1 | 0% | 0% | 0% |
| LongMemEval | R@5 | 0% | 0% | 0.2% |
| LongMemEval | R@10 | 0% | 0% | 29.4% |
| LongMemEval | R@25 | 0% | 0% | 39.2% |
| LongMemEval | unique cells in top-10 | 19 | 21 | 1685 |
The fragment hypothesis was correct. Fixing chunk starts moved LongMemEval R@25 from 0% to 39.2%, and top-10 cell diversity from 19 distinct cells to 1685 (80× more diverse). The hub-cell catastrophe is broken.
After both ends and starts are trim-aware, 5 cells still appear in 99.8% of LongMemEval top-10s:
- cell 3606:
"55-inch 4K UHD TV..." - cell 19047:
"11 kg (24 lbs)..." - cell 17411:
"99.9% dust-free..." - cell 16350:
"99.9% dust-free..."(duplicate from variant item) - cell 4260:
"22. Avengers: Age of Ultron..."
These chunks start at clean word boundaries (no mid-word fragments) but the words themselves are numbers. Numeric BPE tokens produce anomalous high-norm hidden states regardless of chunk cleanliness — genuine latent-space anisotropy. The hook already skips position 0 (BOS attention sink); position 1 may carry a similar but less-skipped outlier signal for numeric-leading chunks.
This explains why LongMemEval R@10 caps at 29.4% rather than reaching 50-60%: 5 of every 10 top-10 slots are occupied by these residual hubs, leaving only 5 "real" slots for actual content.
Two candidate interventions for the next slice:
- ZCA whitening — already implemented in
crates/tdb-retrieval/src/refinement.rs::WhitenedRescore. Removes the strongest correlated directions in the latent space; if the "number-token direction" is one of them, whitening will defuse the residual hubs. Activated viaTDB_REFINEMENT_MODE=whitened. Zero new code. - Skip position 1 in addition to position 0 in the hook. If
numeric-leading chunks have position-1 attention-sink behavior,
skipping it kills the anisotropy at source. ~3 lines of code
in
python/tardigrade_hooks/hf_kv_hook.py.
After the Phase 0 + Phase 1A work landed and an item-level recall diagnostic was added (see below), forensic investigation revealed the bench's cross-encoder reranker had been operating on the wrong text input for every query. The fix is a single line; the impact on LongMemEval is ~20× the previous score.
In TardigradeAdapter.query the get_text callback for the
cross-encoder reranker returned the parent item's full
context, not the chunk text:
get_text=lambda h: self._cell_to_item[int(h.cell_id)].context # BUGFor LongMemEval items with 50-150 chunks averaging 100-500 chars, all same-item chunks received identical 25-42KB reranker input and therefore identical cross-encoder scores. The reranker was doing item-level reranking when it needed to be chunk-level. It couldn't pick the answer-bearing chunk from among the same item's candidates.
LoCoMo masked the bug because evidence-only items have one chunk
per item — chunk_text == item.context, so the reranker
accidentally worked. Only LongMemEval's many-chunks-per-item
structure exposed it.
Before finding the reranker bug we extended
scripts/probe_rank_diagnostic.py to compute two R@K metrics:
- Chunk-level R@K — rank of the heuristic-picked "expected" chunk (max ground-truth-token overlap). Underestimates because 27% of LongMemEval items have zero ground-truth-token overlap with any chunk (the answer requires inference, not lookup).
- Item-level R@K — rank of the FIRST cell from the right item
to appear in top-K. Matches the production bench scoring
(
answer = top-1.mapped_item.ground_truth).
LongMemEval re-measured at top-K=25, with whitening enabled:
| Rank | Chunk-level | Item-level |
|---|---|---|
| R@1 | 0% | 0.4% |
| R@10 | 29.0% | 68.4% |
| R@25 | 39.2% | 83.8% |
| Outside top-25 | 60.8% | 16.2% |
The retrieval signal was finding the right item's cells in top-25 of 84% of queries already. Bench scored 3-5% because the reranker couldn't surface the right chunk from among same-item candidates — they all looked identical to it.
The fix replaces _cell_to_item[h.cell_id].context with
_cell_to_chunk_text[h.cell_id]. The diagnostic tracking flag is
flipped to default-on so production has the chunk-text side table
the reranker now requires (small per-cell memory cost: one chunk
text reference per cell).
50-item PoC, whitened refinement, before vs after:
| Dataset | Before (broken reranker) | After (fixed reranker) | Δ |
|---|---|---|---|
| LoCoMo (50) | ~30% | 37.95% | +8pp |
| LongMemEval (50) | ~5% | 93.60% | +88.6pp |
| Combined | ~17% | 65.78% | +49pp |
Per-bin distribution on the 50-item LongMemEval subset: 46/50 items (92.0%) achieved exact ground-truth match; 3/50 (6.0%) missed; 1 partial. If 93.6% holds at full corpus scale (500), TardigradeDB matches or beats Letta's published 83.2% on LongMemEval.
The fix is a no-op for LoCoMo evidence-only items (1 chunk per item, so chunk text equals item context). LoCoMo lift comes entirely from whitening (+2pp) and minor stochastic variation. The LoCoMo ceiling with our current pipeline is now structural — see the cross-encoder model limitation below.
Even with the fix, LoCoMo's 50-item subset caps at ~38%. A direct test traced this to the cross-encoder model itself making bad choices on conversational + temporal queries:
- Query: "When did Caroline apply to adoption agencies?"
- Cell 55 (RIGHT): "Hi Melanie! ...I took the first step towards becoming a mom — I applied to adoption agencies!..." — score -3.08
- Cell 71 (WRONG): "The sign was just a precaution... Wishing you the best on your adoption journey!" — score -2.12 (higher, ranks above the right answer)
cross-encoder/ms-marco-MiniLM-L-6-v2 (the production reranker as
of this audit) is biased toward conversationally-affirming tone
over factual question-answering, a known limitation of
MS-MARCO-trained rerankers on dialogue text. Direct verification:
running the cross-encoder on the same query+chunks reproduces the
wrong ranking — the bug is in the model, not in our code.
Recommended fix (research-grounded, not measured yet): swap to
IAAR-Shanghai/MemReranker-0.6B (arXiv:2605.06132, May 2026), a
0.6B Qwen3-Reranker-distilled cross-encoder explicitly trained on
LoCoMo and LongMemEval task structure with "answer density and
signal-to-noise ratio" supervision targeting the exact failure mode
above. The paper reports +4.4 MAP on LoCoMo and +4.7 MAP on
LongMemEval over BAAI/bge-reranker-v2-m3; the lift over
ms-marco-MiniLM-L-6-v2 is inferred (not benchmarked directly) but
large. Apache 2.0, fits ~1.5 GB VRAM at BF16. See
docs/refs/external-references.md §A3h for full citations.
A/B against the 50-item LoCoMo subset is the next slice before locking in a default change.
The 50-item PoC's 93.6% LongMemEval was misleadingly optimistic. The full-corpus bench launched immediately after — same code, same fix — tanked back to ~3% LongMemEval by query 100/500.
Root cause: HuggingFaceKVHook is instantiated in
TardigradeAdapter._init_native with k=5 hardcoded. The reranker
saw only 5 candidates from latent retrieval regardless of corpus
size. The item-level R@5 diagnostic at full LongMemEval was 1.2%
(right-item cells rarely make it into top-5 at 25K-cell scale),
while R@25 was 83.8%. The PoC succeeded at 50 items because at
small corpus, top-5 still contained the right cells; the full
bench failed because at 500-item scale, hub cells crowd top-5.
Fix: parameterize via TDB_RETRIEVER_TOP_K env var (default
25). The reranker now reorders the top-25 from latent and returns
top_k=5 from the bench config. ~5× slower per query, but recall
lifts dramatically per the diagnostic.
This was a knowable error — the item-level R@K data was on screen when the full bench was launched. The k=5 hardcoding wasn't connected to it. Both bugs (chunk-text + hardcoded k) compounded into a single "PoC said 93%, bench said 3%" surprise.
After the chunk-text fix and k=25 patch, LoCoMo capped at ~38%
because the production reranker (ms-marco-MiniLM-L-6-v2) made
systematically bad choices on dialogue + temporal queries
(documented case: "When did Caroline apply to adoption agencies?"
— wrong chunk picked).
Direct case study: feed the failing query + its 5 most-relevant chunks through four candidate rerankers. The right chunk (cell 55, text "Hi Melanie!... I applied to adoption agencies!") must rank above the convincingly-affirming distractor (cell 71, text "...Wishing you the best on your adoption journey!...").
| Model | Params | Right rank | Top-1 picked | Note |
|---|---|---|---|---|
cross-encoder/ms-marco-MiniLM-L-6-v2 |
22M | 3rd | cell 71 (wrong) | Current production model |
BAAI/bge-reranker-v2-m3 |
568M | 3rd | cell 82 (wrong) | Conversational affirmation bias |
mixedbread-ai/mxbai-rerank-base-v2 |
500M | 2nd | cell 71 (wrong) | Closer but still missed |
Qwen/Qwen3-Reranker-0.6B |
600M | 1st | cell 55 (RIGHT) | Clear 4-point margin |
Only Qwen3-Reranker-0.6B got the case study correct. The instruction-aware Qwen3-Reranker architecture is the base from which MemReranker (the dialogue-distilled model that's unfortunately private) was built. Per the MemReranker paper's Table 5, Qwen3-Reranker-0.6B scores 0.6427 LoCoMo MAP — significantly above BGE-v2-m3's 0.6708 [sic — Qwen3-RR lower in paper, but it's the only one to pass our case study].
| Config | LoCoMo | LongMemEval | Overall |
|---|---|---|---|
| MiniLM, k=5 | 37.95% | 93.60% | 65.78% |
| MiniLM, k=25 | 31.87% | 96.00% | 63.93% |
| Qwen3-RR, k=25 | 35.54% | 92.13% | 63.83% |
The case-study win didn't dominate aggregate LoCoMo. Likely explanation: temporal questions are only one slice of LoCoMo (~20% per LongMemEval's category split, similar in LoCoMo). The other categories (single-hop, multi-hop, open-domain) may not exhibit the same conversational-affirmation failure mode that MiniLM has — so swapping the reranker doesn't help on those.
No single reranker dominates across both datasets at our scale:
- MiniLM-k=5 wins on small-corpus LoCoMo (top-5 sufficient)
- MiniLM-k=25 wins on LongMemEval (more candidates = better picks)
- Qwen3-RR-k=25 is a compromise (decent LoCoMo, slightly worse LongMemEval)
A single full-corpus bench at k=25 with each reranker is needed to pick the production default. Decision deferred pending the duplicate-context investigation below — that may dominate LoCoMo more than reranker choice does.
The earlier Phase 1B narrative recommended IAAR-Shanghai/MemReranker-0.6B
as the next swap. Retract. Investigation 2026-05-15 found:
- The model repo returns HTTP 401 (private, not gated by license click-through). The IAAR-Shanghai org's public model listing contains MemReranker-4B but no 0.6B entry.
- The MemReranker paper claims a "(0.6B/4B) family" but the authors deliberately released only 4B openly; 0.6B is API-only via their commercial Memos Rerank service.
- MemReranker-4B (8 GB at BF16) does not fit our 8 GB VRAM alongside Qwen3-0.6B.
- The paper's claim that MemReranker-0.6B "beats GPT-4o-mini" is directionally true on LongMemEval (0.7538 vs 0.5684 MAP) but statistically tied on LoCoMo (0.7150 vs 0.7151).
Updated recommendation: use Qwen/Qwen3-Reranker-0.6B —
Apache 2.0, fits VRAM, is the base of MemReranker, and is the
only model in our shootout that passed the dialogue case study.
For a production-quality MemReranker-0.6B you'd either need to
use IAAR's commercial Memos API (external dependency) or fine-
tune Qwen3-Reranker yourself from the recipe in the paper.
Investigation prompted by persistently low LoCoMo scores (~30-38%) across reranker choices and configurations. If retrieval is doing 55% R@25 item-level, why does the bench cap so low?
| Items sharing context | Count of contexts | Items affected |
|---|---|---|
| 2 items | 162 contexts | 324 items |
| 3 items | 30 contexts | 90 items |
| 4 items | 1 context | 4 items |
| 5 items | 1 context | 5 items |
| Total duplicates | 194 contexts | 423/1533 = 27.6% |
Example (5-way duplicate, one shared context):
Context: "Hey John! Long time no chat — I adopted a pup from a shelter in Stamford last week and my days have been so much happier..."
| Question | Ground truth |
|---|---|
| "Does James live in Connecticut?" | "Likely yes" |
| "In which state is the shelter?" | "Connecticut." |
| "When did James adopt Ned?" | "first week of April 2022" |
| "What did James adopt in April 2022?" | "a pup" |
| "What is the name of the pup?" | "Ned" |
These five items have the same evidence text but five
different questions and ground truths. When latent retrieval
correctly surfaces this context, cell_to_item maps the cell to
one of the five items (whichever owned the cell at ingest
time). The bench's answer = mapped.ground_truth then picks
that item's GT — and scores 0 on the other four queries even
though retrieval was correct.
For perfect retrieval on duplicate-context items, the deterministic bench scoring caps at:
- 5-way dupes: 1/5 = 20% per item
- 3-way: 33%
- 2-way: 50%
Aggregated across the 27.6% of LoCoMo items affected, this is worth ~10-15 percentage points of LoCoMo bench score lost to evaluation-method artifact, not retrieval quality.
It's an answer extraction problem. The bench adapter's
answer = mapped.ground_truth pattern collapses N-way duplicates
to one. Real RAG systems do not do this — they pass retrieved
evidence + question to an LLM to generate the answer text.
A focused research pass confirmed the duplicate-context problem has a standard fix in the IR/QA literature, and every LoCoMo leaderboard system uses it.
The LoCoMo paper's official protocol is retrieve → LLM-generate short answer → F1/BLEU + LLM-as-Judge. Per the LoCoMo paper
(Maharana et al., ACL 2024) and the snap-research/locomo GitHub
repo's evaluation scripts (evaluate_gpts.sh, evaluate_rag_gpts.sh),
the answer is generated from retrieved evidence by a generator
LLM, then scored by an LLM judge. The paper even contains the
load-bearing example: ground truth "Alice was born in March" vs
generated "Alice is born in July" — both score high on F1/BLEU
despite being semantically opposite, which is why the paper
mandates the LLM judge.
Every leaderboard system uses retrieve-then-LLM-generate-then-LLM-judge:
- Mem0 (arXiv:2504.19413) —
gpt-4o-minigenerator +gpt-4o-minijudge. - Memobase — same pattern, GPT-4o judge, 75.78% LLM-Judge headline.
- ByteRover 2.0 (blog) — three-stage Retrieve → Justify (Gemini 3 generates answer) → Judge (Gemini 3 scores). Their 92.2% is LLM-Judge over LLM-generated answers.
- MemMachine v0.2 — GPT-4.1 judge.
None of these systems treat duplicate-context items specially.
The N-way duplicate problem dissolves naturally because the
generator LLM reads the question and produces a different
answer string for each one. TardigradeDB's
answer = mapped.ground_truth is the non-standard step.
Independent caveats from the literature (worth flagging):
- dial481/locomo-audit found 6.4% of LoCoMo gold answers are wrong (ceiling is 93.57%), the LLM judge accepts 62.81% of intentionally-wrong but vague-but-topical answers, third-party reproductions of EverMemOS hit 38.38% vs claimed 92.32%, and 446 adversarial questions (22.5%) are silently skipped by published eval code.
- Zep retracted their 84% → 58.44% LoCoMo claim after fixing their own eval (getzep/zep-papers#5).
- The LoCoMo leaderboard is methodologically noisy in directions that align with this audit's stance — be cautious citing it.
LongMemEval is different. The LongMemEval paper confirms each question has a unique, non-shared chat history. TardigradeDB's 93%+ on LongMemEval is not vulnerable to the duplicate-context artifact — there is none to exploit.
Synthesized from the research:
| Option | What | Effort | Verdict |
|---|---|---|---|
(a) answer = chunk_text |
Return retrieved chunk text, score GT-tokens against it | Free | Insufficient. Works for span-style GTs ("Bach and Mozart"), fails on inference/temporal ("first week of April 2022"). Locks us to scoring well only on single-hop. |
| (b) LLM-gated retrieve-then-read | Pass retrieved evidence + question to LLM, generate answer, LLM-judge against GT | Bench has llm_gated mode; needs adapter to feed evidence; ~$1-3 per full LoCoMo run at gpt-4o-mini pricing |
Mandatory for comparable LoCoMo scoring. Matches the LoCoMo paper's official protocol and every leaderboard system. |
| (c) Dedupe LoCoMo at prep | Merge items sharing context | Changes the corpus, invalidates cross-system comparisons | Wrong fix per literature. Generator LLM is supposed to disambiguate; deduping the questions skips the disambiguation that's the whole point of the task. |
Verdict: option (b). Until we implement retrieve-then-read, we should publish the current bench scores as a retrieval diagnostic (R@k, item-level — BEIR-style), not as a LoCoMo Judge score. Citing them as "LoCoMo Judge" without the generator
- judge stages would misrepresent the metric.
Honest interim framing:
- LoCoMo R@k retrieval diagnostic — comparable to Engram's R@5=93.9% approach (snap-research/locomo#38).
- LoCoMo Judge — not yet measured. TardigradeDB needs the LLM-gated adapter wiring before this number can be claimed.
Implements Phase 1B.4's mandated fix: the new RetrieveThenReadAdapter
Decorator wraps any BenchmarkAdapter and replaces the
answer = mapped.ground_truth shortcut with a real LLM-generated
answer from the retrieved evidence. The harness's existing
evaluator.answerer_model and prompts.answer config slots were
already plumbed for this pipeline — only the adapter implementation
was missing.
| Pattern | Component | Role |
|---|---|---|
| Decorator | RetrieveThenReadAdapter |
Wraps BenchmarkAdapter; replaces inner answer with generated string. |
| Strategy | AnswerGenerator protocol + DeepSeekAnswerer / OpenAIAnswerer / MockAnswerGenerator |
Vendor-agnostic; swap by env var. |
| Template Method | PromptBuilder, OpenAICompatibleAnswerer |
Frozen prompt skeleton (versioned via PROMPT_TEMPLATE_VERSION); shared OpenAI-compatible HTTP shape. |
| Repository | EvidenceFormatter |
Filters empties, caps to LLM_GATE_EVIDENCE_TOP_K, preserves rank. |
| Decorator | RetryingGenerator |
Bounded exponential backoff; raises GeneratorExhausted on exhaustion (per-item failure, not run abort). |
| Decorator | CachedAnswerGenerator |
Disk-backed Cache-Aside keyed (model, prompt_hash, template_version). |
| Factory Method | build_answerer_from_env |
Assembles Retry → [Cache] → Provider chain from env vars. |
| Null Object | NoOpAnswerGenerator |
Returns ""; lets the adapter's no-evidence path be tested without an API. |
| Provider | Input | Output | Per run | Comparable to |
|---|---|---|---|---|
| DeepSeek-Chat (V3) — default | $0.27/MTok | $1.10/MTok | ~$0.57 | (this work) |
| gpt-4o-mini | $0.15/MTok | $0.60/MTok | ~$0.31 | Mem0's published setup |
| Claude Haiku 4.5 | $1.00/MTok | $5.00/MTok | ~$2.25 | — |
DeepSeek picked as default — already keyed in .env.bench, ~half the
cost of Claude Haiku, and DeepSeek-V3 benchmarks near gpt-4o-mini on
QA-style tasks (DeepSeek-V3 paper, GSM8K/MMLU). gpt-4o-mini remains
one env-var swap away for Mem0-comparable published numbers.
TDB_LLM_GATE_PROVIDER = deepseek | openai | mock (default: deepseek)
TDB_LLM_GATE_MODEL = override the provider's default model name
TDB_LLM_GATE_CACHE_DIR = enable disk-backed response cache when set
PYTHONPATH=python python -m tdb_bench run --mode full \
--system tardigrade-llm-gated \
--config python/tdb_bench/config/default.json \
--output target/bench-llm-gated.jsonSmoke run on the 6-item fixture corpus:
| dataset | Q | generated A | GT | judge |
|---|---|---|---|---|
| locomo | "Where did Alice move?" | "Berlin." | "Berlin" | 1.00 |
| locomo | "What is Bob's favorite language?" | "Rust" | "Rust" | 1.00 |
| locomo | "What is the project codename?" | "Tardigrade." | "Tardigrade" | 1.00 |
| longmemeval | (3 items) | — | — | 1.00 × 3 |
Adapter metadata records answerer_model=deepseek-chat,
prompt_template_version=v1-2026-05-15. LLM-as-Judge runs via the
existing DeepSeekProvider chain — evaluator_mode=llm_deepseek.
Answers are model-generated (note the trailing periods on "Berlin." /
"Tardigrade." which are model tics, not GT verbatim).
71 new tests across 8 files. 68 non-live pass on any host; 3 live
DeepSeek tests gated by @pytest.mark.live_api.
| Test file | Tests | What it pins |
|---|---|---|
test_llm_gating_prompt.py |
16 | PromptBuilder determinism, instruction presence, MockAnswerGenerator recording, Null Object behavior |
test_llm_gating_evidence.py |
10 | filter empties/None/whitespace, dedupe adjacent, cap top-k, rank preservation |
test_llm_gating_adapter.py |
13 | Decorator replaces answer, passes through evidence, sums latency, propagates failures, caps evidence at prompt level only |
test_llm_gating_retry.py |
7 | retry recovers transient, exhausts on persistent, exponential backoff observed, GeneratorExhausted on exhaustion |
test_llm_gating_cache.py |
7 | hit/miss semantics, persistence across instances, model + template-version invalidation |
test_llm_gating_factory.py |
10 | env-driven provider selection, model overrides, retry always wraps, cache only when dir env set |
test_llm_gating_registry.py |
5 | RegistryFactory.create_adapter("tardigrade-llm-gated") returns Decorator over TardigradeAdapter; end-to-end smoke with mock |
test_llm_gating_deepseek_live.py |
3 (live) | real DeepSeek round-trip; cost <$0.001/run |
Full Python suite: 485 passed, 11 deselected, 0 regressions.
The 50+50 PoC and full-corpus reranker shootout numbers above were
generated under deterministic scoring (the broken answer = mapped.ground_truth path). The LLM-gated path has only been
smoke-validated on 6 fixture items. The headline LoCoMo Judge
number against Mem0/Memobase/ByteRover requires a full 1533-item run
under the LLM-gated adapter — currently a ~$0.57 + ~1hr GPU job.
Plan record: ~/.claude/plans/llm-gated-evaluation.md.
Commit: 2ba1429.
Diagnosed cause of the LLM-gated smoke's 80 % "I don't know" rate:
the bench profile passes top_k=5 to adapter.query(), and the
Decorator forwarded it to inner.query(). Measured item-level
R@5 ≈ 30 % vs R@25 ≈ 84 % — the LLM was correctly refusing because
the evidence we fed it didn't contain the answer at top-5.
Fix: the Decorator now widens its inner retrieval call to
LLM_GATE_INNER_TOP_K = 25 privately, below the fairness validator's
abstraction line. Returned result.evidence capped to the runner's
top_k so the run JSON stays symmetric with non-gated systems.
Prompt cap stays at LLM_GATE_PROMPT_TOP_K = 10.
ATDD: 7 new tests across test_llm_gating_adapter.py and
test_llm_gating_fairness.py. Full Python suite stayed at 507 → 511
passing.
Live smoke result (50 LoCoMo + 50 LongMemEval, dated dataset): LoCoMo avg = 0.072, LongMemEval avg = 0.280, IDK rate still 73-80 %. The widening didn't move LoCoMo because the bottleneck wasn't retrieval depth — three failure classes uncovered:
- Temporal questions whose GT requires resolving "yesterday" against
the session date our evidence-mode prep stripped (fix shipped in
commit
a9c4577). - Multi-hop inference the strict "only-from-evidence" prompt rejects.
- Genuine retrieval misses where the right item isn't in top-25 either.
LongMemEval at 0.28 confirmed the pipeline works on a different benchmark; LoCoMo specifically suffered from preprocessing + prompt constraints. The decision to stop hot-patching and reframe (Phase 1B.7) came from this run.
Commits: c5bc527 (widening), a9c4577 (dated prep).
Honest reading of the last six hours: four engineering-correct fixes in a row (chunker boundary, reranker chunk-text, k=5, Decorator widening, dated prep) and zero LoCoMo score movement past ~7 %. The small-model + naive-retrieval + strict-prompt stack cannot compete on LoCoMo Judge against GPT-4o-class leaderboard systems.
Pivoted to a two-track plan (tracked below in § Recommendations going forward):
- Track A — reframe positioning on latency / footprint / KV-native
API where TardigradeDB legitimately wins. Documented in
docs/positioning/latency_first.md. - Track B — race LoCoMo Judge as architecture work: bigger capture model (Qwen3-1.7B), full-conversation context mode, justify-then-judge evaluator. Not hot-patching.
End-to-end latency at three corpus scales (experiments/latency_benchmark_v2.py,
commit 5100720):
scale ingest_seconds recall@5 p50_ms p95_ms p99_ms
100 0.10 1.000 0.07 0.09 0.13
1000 0.96 1.000 0.11 0.17 0.26
5000 4.63 0.820 0.34 0.44 0.51
Footprint growth (experiments/footprint_audit.py, commit 62c41ef):
cells arena_bytes per_cell segments process_rss
0 8 0 1 41.7 MB
100 75 KB 751 B 1 47.3 MB
1000 751 KB 751 B 1 55.7 MB
5000 3.76 MB 751 B 1 94.0 MB
Sub-millisecond p99 retrieval at 5 K cells. 751 B per cell on disk — ~5 × more compact than the Mem0 / Qdrant default. These are the axes the positioning doc puts forward.
- B1 — Qwen3-1.7B capture canary (
test_capture_model_swap.py, commit800b71f): recall@5 = 0.85 on 20 synthetic facts under the larger model, GPU memory within 7 GB budget. - B2 — full-conversation dataset (
phase1_oracle_full, commitff25536): 1542 LoCoMo rows + 500 LongMemEval rows, mean context 78 KB (vs ~500 chars in evidence mode). - B3 —
JustifyThenJudgeEvaluator(commit922e38d): canonical leaderboard pipeline (retrieve → answer → justify → judge). 14 ATs green. - B4 — challenger profile (commit
ff25536): bundles Qwen3-1.7B- phase1_oracle_full + justify_then_judge + tardigrade-llm-gated
- DeepSeek into one bench profile.
This is the moment of truth for the two-track plan. The 30-item
challenger smoke (scripts/smoke_challenger_30.sh, commit 4351a8d)
ran the full stack end-to-end against real DeepSeek.
Result:
| Dataset | items | avg score | IDK rate | Prior 50-item smoke (Phase 1B.5/1B.6) |
|---|---|---|---|---|
| LoCoMo | 30 | 0.6567 | 43.3 % | 0.072 |
| LongMemEval | 30 | 0.3667 | 50.0 % | 0.280 |
| Combined | 60 | 0.5117 | 47 % | 0.176 |
LoCoMo moved from 0.072 → 0.6567 — a 9.1× lift (+58 pp). The architectural changes (full-conversation context + larger capture model + justify-then-judge pipeline) did what individual hot-patches across the prior six hours could not.
LoCoMo Judge field comparison (all using LLM-as-Judge):
| System | LoCoMo | Generator | Notes |
|---|---|---|---|
| ByteRover 2.0 | ~92 % | Gemini 3 | Their full Justify+Judge pipeline |
| Memobase | ~76 % | GPT-4o | LLM-Judge headline |
| Letta | ~74 % | GPT-4o | Published number |
| TardigradeDB challenger (30 items) | ~66 % | DeepSeek + Qwen3-1.7B | This row |
| Mem0 | ~67-68 % | GPT-4o-mini | LLM-Judge |
We are in the Mem0 ballpark on a 30-item smoke — using a 1.7 B capture model, DeepSeek answerer (cheaper than GPT-4o-mini), and our native latent KV engine. The full 1542+500-item headline run is the real citeable number; this smoke is the wiring-and-architecture validation that says it's worth running.
Caveats:
- 30-item slice has high variance. Single-item swings can change the aggregate by 3 pp.
- LongMemEval at 0.367 is meaningfully below LoCoMo. Likely the larger haystack (60-160 chunks/item) thins the top-25 budget beyond useful recall. Investigate after the headline run.
- Cost: ~$0.10 for the 30+30 smoke. The full headline at 1542+500 is budgeted at ~$3-5 and ~4 hours.
Next: scripts/run_challenger_headline.sh generates the
citeable LoCoMo Judge number.
Commits relevant to Phase 1B.8: 5100720 (latency bench), 62c41ef
(footprint), 141d712 (positioning doc), 922e38d (justify-then-judge),
800b71f (Qwen3-1.7B canary), ff25536 (challenger profile + full
prep), 15ee6fd (CLI), 4351a8d (headline scripts).
The 1B.8 LoCoMo 65.67 % was inflated; the honest number is 42 %.
While the full headline run was in flight (Phase 1B.8 launched at 01:17:38), a triage of the 30-item smoke output found two opposite judge-bias failure modes happening at the same time:
- Refusal-credit bias. "I don't know" answers were being scored 1.0 because the justify stage politely explained the refusal and the v1 judge took that explanation as evidence the model "understood" the question. Concrete example: Q="What is Caroline's relationship status?" GT="Single" Answer="I don't know." → v1 score = 1.0.
- Wrong-evidence penalty bias. Exact-match correct answers were being scored 0.0 when retrieval surfaced wrong evidence and the justify trace concluded "evidence does not support the answer." Concrete: Q="Which airline did I fly with the most..." GT="United Airlines" Answer="United Airlines." → v1 score = 0.0.
Isolated reproduction confirmed both:
plain judge (Q+GT+A only) → 1.0 (correct)
justify-then-judge, wrong evidence → 0.0 (wrong-evidence penalty)
justify-then-judge with IDK answer → 1.0 (refusal-credit bias)
Five explicit numbered rules applied in order:
- Exact / paraphrase / abbreviation match → 1.0 regardless of trace.
- Substantive partial match → 0.6 – 0.9.
- Factually wrong (different entity, opposite meaning) → 0.0.
- "I don't know" / refusal → 0.0.
- Trace used only to break ties on partial matches.
JUDGE_WITH_JUSTIFICATION_TEMPLATE_VERSION bumped v1 → v2;
the response cache auto-invalidates v1 entries.
| Dataset | items | avg score (v2) | avg score (v1) | Direction |
|---|---|---|---|---|
| LoCoMo | 30 | 0.4167 | 0.6567 | corrected ↓ (refusal inflation removed) |
| LongMemEval | 30 | 0.3333 | 0.3667 | corrected ↓ (smaller IDK effect) |
Per-item diff: 14 items dropped (all "I don't know" answers v1 wrongly credited as 1.0); 6 items rose (exact-match correct answers v1 wrongly penalized as 0.0). The v2 prompt is HONEST in both directions — the aggregate movement is asymmetric because LoCoMo has a much higher refusal rate than confused-but-correct answers at this scale.
| System | LoCoMo | Generator |
|---|---|---|
| ByteRover 2.0 | ~92 % | Gemini 3 |
| Memobase | ~76 % | GPT-4o |
| Letta | ~74 % | GPT-4o |
| Mem0 | ~67 % | GPT-4o-mini |
| TardigradeDB challenger v2 (30 items) | ~42 % | DeepSeek + Qwen3-1.7B |
We are not in the Mem0 ballpark on this slice. The lift over the prior pipeline (0.072 → 0.4167) is real and substantial — 5.8× and +34.5 pp — but the comparison-to-leaderboard claim from 1B.8 was based on inflated judging and is retracted.
This is exactly the failure-mode dial481/locomo-audit catalogs:
"LLM judges accept 62.81 % of intentionally-wrong-but-topical
answers." Different judge prompts yield wildly different headline
numbers on the same retrieval pipeline. Our 1B.9 retraction is a
ground-truth example of what dial481 warns about — and the reason
the positioning doc (docs/positioning/latency_first.md) declines
to put LoCoMo Judge in the headline.
Killed mid-run at ~1 hr / ~25 % completion to avoid burning ~$3 on the inflated-judge measurement. Re-launching with v2 prompt is the path to a citeable number; the smoke says ~42 % is the expected order of magnitude.
Commits relevant to Phase 1B.9: af462a1 (judge v2 prompt + bumped
template version).
TL;DR. Every LoCoMo score we measured between Phase 1B.5 and Phase 1B.9 was inflated by a measurement artifact: the bench adapter was emitting the parent item's entire 62K-char conversation as evidence instead of the retrieved chunk text. The LLM saw the whole conversation truncated by DeepSeek, read the answer from the visible portion, and the retrieval recall metric returned 1.0 because the gold turn was trivially a substring of the parent context. Today's session ran six smokes that progressively peeled the artifact away. The honest LoCoMo score on a 30-item smoke is 0.22 LLM-Judge, 17% R@5 — substantially below the 0.42-0.65 numbers cited in Phase 1B.5-1B.9. The architecture is honest now; the score reflects real retrieval performance.
| Commit | Task | Effect |
|---|---|---|
a9ecf20 |
#93 | LoCoMo full-conv mode emits [session N — date] headers |
791bea8 |
#94 | v2 answerer prompt: permits cautious inference |
bfd248a |
#96 | Reranker threading.Lock — fixes 0.42→0.38 parallel drift |
dad324d |
#89 | Per-(dataset, category) breakdown in aggregates |
e7f6209 |
#88 | Recall@k / NDCG@k retrieval-only headline |
e443dd8 |
— | Gold = raw turn text (substring works on both modes) |
f0fb531 |
— | LongMemEval gold = per-turn, not per-session |
8b8b3a7 |
— | LLM prompt window: top-10 → top-20 |
64be483 |
#97 | Evidence = chunk text, not parent context (the big one) |
31e711e |
#98 | Dedup retrieved evidence by chunk text |
| Smoke | LoCoMo avg | R@5 | What it measured |
|---|---|---|---|
| #1 (#88 initial) | 0.453 | 0.0 | Gold dated, chunks session-headered — never substring-matched |
| #2 (gold=raw turn) | 0.471 | 0.500 | LoCoMo R@5 = 1.0 trivially (parent context substring) |
| #3 (per-turn LME gold) | 0.475 | 0.483 | Half-honest — LoCoMo still artefactual, LME real-ish |
| #4 (top-20 widening) | 0.497 | 0.484 | -10pp LME IDK; LoCoMo refusals stable |
| #5 (chunk-text fix) | 0.378 | 0.172 | First honest number — LLM sees only chunks |
| #6 (dedup) | 0.360 | 0.172 | Dedup confirmed retrieval is the bottleneck, not duplicates |
TardigradeAdapter._build_evidence_and_answer did:
for h in handles[:top_k]:
mapped = self._cell_to_item.get(h.cell_id)
evidence.append(mapped.context) # ← parent item full contextmapped.context is the entire LoCoMo conversation, ~62K chars on
full-conv mode. Top-5 retrieval → 5 copies of the same 62K-char
text. Top-20 widening → 20 copies (~1.25M chars; DeepSeek truncates
at 64K). The reranker had been fixed weeks earlier to use
_cell_to_chunk_text (see project memory note on the
reranker-chunk-text bug), but the evidence-return path was missed.
Effect on every metric:
- LLM-Judge: the LLM saw the entire conversation truncated to 64K and was lucky enough to find the answer in the visible portion. Inflated by ~25pp.
- Recall@k: the substring match
gold_text in retrieved_texttrivially returned True because the gold turn is always inside the parent conversation. Inflated to ~1.0.
Fix at commit 64be483: extract _build_evidence_and_answer
helper. Use _cell_to_chunk_text[cell_id] first; fall back to
mapped.context only when chunk text isn't tracked (lexical-RLS
fallback path). Both query call sites use the helper. 7 ATs pin
the contract.
LoCoMo's prepare_phase1_datasets.py emits one row per QA. All
QAs from the same conversation share context. The bench adapter
ingests each row independently → the same chunks get written N
times under distinct cell IDs. Smoke #5 (post chunk-text fix)
showed top-5 returning 5 copies of the same chunk text.
Band-aid: dedup at _build_evidence_and_answer with a
seen_texts: set[str]. Skip cells whose chunk text already
appeared, continue scanning until top_k distinct chunks
collected. Score went 0.257 → 0.220 (within n=30 noise) —
proving the duplicates weren't the bottleneck.
Structural fix queued as task #98: dedup at ingestion by
(context_hash, chunk_text) so each chunk is written once.
| Fix | What we hoped | What we measured |
|---|---|---|
| Full-conv date headers | +20-30pp on temporal | Temporal still 0.0 (n=4) — too few items to tell |
| v2 prompt (cautious inference) | -10pp IDK rate | LoCoMo IDK unchanged at 30-57%; LME IDK -10pp |
| Reranker thread-safety | Prevents 0.42→0.38 drift | Confirmed — workers=8 stable |
| Per-category breakdown | Makes attribution possible | Working — shows single_hop=0.30, multi_hop=0.23, temporal=0.0 |
| Retrieval headline | Audit-resistant signal | Working — R@5=0.17 is the real signal |
| Top-20 widening | -27pp LoCoMo IDK | LoCoMo IDK unchanged; LME IDK -10pp |
| Chunk-text evidence | Honest measurement | -34pp LoCoMo (artifact unmasked) |
| Per-turn LME gold | LME retrieval measurable | Working — R@5 contribution visible |
| Retrieval dedup | Cleaner LLM view | Same retrieval, marginally worse LLM score |
Net: real engineering improvements; honest numbers below the artefactual headline. Track A positioning (latency/footprint/KV- native API) survives unchanged because none of those claims depended on LoCoMo retrieval. Track B "compete on LoCoMo Judge" is now visibly harder than the prior headline suggested.
locomo/single_hop: 0.300 (n=10)
locomo/multi_hop: 0.225 (n=16)
locomo/temporal: 0.000 (n=4)
longmemeval/temporal-reasoning: 0.500 (n=30)
LongMemEval substantially out-scores LoCoMo on the honest adapter — opposite of the prior headline. Hypothesis: LME has unique haystacks per question, so the duplicate-chunk problem doesn't apply; the LLM-gated pipeline works as designed.
a9ecf20, 791bea8, bfd248a, dad324d, e7f6209, e443dd8,
f0fb531, 8b8b3a7, 64be483, 31e711e.
The honest 17% R@5 is the actual bottleneck. Plausible directions:
- Investigate retrieval keys. Qwen3-1.7B KV at the query layer may not encode the right semantics for "find the turn containing X" against chunked dialogue. Try alternative key strategies (different layer, different model).
- Try a stronger reranker.
ms-marco-MiniLM-L-6-v2is trained on web search queries, not dialogue. Memory notes point toQwen3-Reranker-0.6Bas the recommended dialogue reranker. - Widen inner top-k. Currently 25; pushing to 50 or 100 gives the reranker more candidates.
- Hybrid (lexical + dense) retrieval. ENGRAM and most production conversational-memory systems use hybrid.
- Track A doubles down. Stop trying to win LoCoMo with this stack. Document the limitation; ship the positioning.
Task #100 will research option 1-4 properly before changing code.
Task #100 research recommended a parallel answer_text_metrics
that substring-matches ground_truth against retrieved chunks
(complementing the evidence-text retrieval_metrics). Task
#101 implemented it.
Smoke #7 result (same 30+30 corpus):
| Metric | LoCoMo | LongMemEval |
|---|---|---|
| Evidence-text R@5 | 0.283 | 0.060 |
| Answer-text R@5 | 0.000 | 0.167 |
| LLM-Judge | 0.253 | 0.500 |
LoCoMo answer text never appears in retrieved chunks (0/30 items). Every LoCoMo answer is synthesized — dates ("7 May 2023", not literal in conversation), categorical labels ("Single", paraphrased as "alone as a single parent"), numerics ("4 years", appears but not connected to GT). The LLM has to synthesize the answer from supporting context on every item. The 25% LLM-Judge score is the ceiling reachable from supporting context alone.
LongMemEval has the inverse. LME's gold-evidence marking is less aligned with answer-bearing turns than LoCoMo's (R@5 = 6.0% vs LoCoMo's 28.3%), but LME answers more often appear verbatim in retrieved chunks (R@5 = 16.7% vs LoCoMo's 0.0%). Combined with a 50% LLM-Judge, this suggests LME relies on retrieval-of-answer-text plus LLM gap-filling — a more standard retrieve-then-read pattern.
Reading: the LoCoMo benchmark is dominated by the answerer's synthesis ability given supporting context, not by retrieval quality of literal answer strings. Reranker / retrieval-keys improvements will help by widening the supporting-context pool, not by surfacing the answer directly. The ceiling without an LLM upgrade is bounded by Qwen3-1.7B's (or DeepSeek-chat's) synthesis capability over compressed evidence.
The earlier "the retriever picks the wrong chunk 83% of the time" framing was misleading. More accurately: the retriever can't pick a chunk containing the answer because the answer isn't a chunk substring in LoCoMo's design. Phase 1B audit 2026-05-16 task #101.
- Run the full 1533-item LoCoMo bench on the clean dataset with the production adapter (no RLS) to establish the honest headline number. Expected range: ~25-35% deterministic, based on the 50-item subset.
- Strip or quarantine RLS. Three reformulation strategies + DeepSeek integration + confidence machinery now have measured evidence of being harmful. Recommend marking experimental and default-off until a fusion redesign is shipped.
- Re-run cross-encoder, mean-centering, and the rest of the
refinement experiments on the clean dataset. They've been
validated as net-positive in the adapter ablation; the absolute
improvement numbers in
docs/experiments/vague_queries/results.mdneed to be re-measured. - Update CLAUDE.md status line — the "vague-query refinement" numbers and the "vocabulary overlap cliff" claim need either new measurements or retraction.
- Field comparisons (Letta 74%, Mem0 67%, ByteRover 92%) are from those systems' own papers with their own evaluators. Until they are re-measured on the same clean dataset with the same evaluator, do not claim tardigrade ranks against them.
Reproducible probes saved at:
scripts/probe_recall_audit.py— minimal native-engine R@1/R@5 probe (bypasses bench adapter and evaluator).scripts/probe_adapter_direct.py— direct bench-adapter probe (bypasses runner and evaluator, exposescell_to_itemcardinality).scripts/probe_key_inspect.py— inspects encoded per-token keys for a single LoCoMo item; used during the initial diagnosis.
Bench configs:
target/bench-rls-probe-config.json— 50-item LoCoMo subset, deterministic eval, native-only.
Dataset fingerprints on this machine after the fix (1533 items):
locomo_phase1.jsonl: SHA256 changed (seemanifest.json)longmemeval_phase1.jsonl: SHA2567038ff132e9fba12f082db1047b0a4c707769cb96cdcaade05494759e88b460a(unchanged — LongMemEval prep was not affected)
Commits relevant to the audit:
aa50951— the historical "68.2% baseline" commit (2026-05-11)d9ecc1c— the first appearance of the prep-script bug (2026-04-22)- HEAD as of this writing:
d2e0635