Route reflection mapped rows through the uniform dedup/merge pipeline

[gorkem2020]

What

Routes reflection mapped rows (User model deltas, Agent model deltas, Lessons, Decisions) through the same dedup/merge pipeline extraction candidates use, whenever smart extraction is enabled.

• New SmartExtractor.persistGatedCandidates(items, options): an entry point for candidates whose extraction and admission already happened in another lane. From there they take exactly the extraction candidates' path: batched dedup decider, verdict handling, batched merge writer, bulk create. A duplicate mapped row now MERGES into its existing target instead of landing beside it.
• Each item supplies its own store-entry builder (a WeakMap keyed by candidate object identity), so a CREATE verdict persists the caller's entry with reflection metadata intact, while merge/skip verdicts operate on existing rows through the shared machinery. Extraction's own candidates can never collide with an external lane's builders.
• The reflection-lane admission gate (gateMappedReflectionEntries) is unchanged and still runs first; a synthetic pass_to_dedup evaluation tells processCandidate not to score admitted rows a second time.
• The no-extractor fallback keeps the direct bulkStore path, strictly exclusive with the new route. Its near-duplicate pre-check is kept but downgraded from fail-closed to fail-open: previously a vector-search failure silently dropped the row; now the row stores, and the worst case is a duplicate a later burst merges.
• Dedup judge failures likewise resolve to create (the batch decider already does this), so a transient LLM outage never loses a reflection cycle.

Why

#931 declared this debt explicitly ("deliberately not reused here yet"): mapped rows got a single 0.95 vector-similarity cutoff as their only dedup, with no LLM-mediated merge/contextualize decision, so reworded repeats accumulate as near-duplicate rows over time. With the batch subsystem from #941 on master, the uniform route costs one batched dedup call per admitted burst (chunked past 10) and zero calls when nothing similar exists.

Notes

• MERGE updates the target row in place, so mapped-row provenance metadata (_reflectionHeading and friends) survives on CREATE verdicts; content merged into an existing row adopts that row's metadata. Importance on non-create verdicts comes from the existing row.
• The dist rebuild also picks up the retriever change from #986, which merged without a dist refresh; the freshness CI requires it to ride along.
• Tests: a new uniform-pipeline suite (merge-into-target, create with metadata intact, one batched dedup call per burst with skip verdicts, judge-failure fail-open, no admission re-scoring for pre-gated rows), plus the existing mapped-rows production-pipeline cells updated to pin both routes (the extractor route never calls bulkStore directly; the no-extractor fallback keeps its historical shape). All five new cells fail on master before this change.

Live evidence

Deployed on a downstream install and exercised with a real reflection run over a day-long session. The burst produced 7 created, 2 merged, 4 skipped: the admission gate independently rejected one row before the pipeline (utility veto), and both merges landed on their existing target rows instead of creating duplicates beside them. No errors across the plugin restart or the run.

[gorkem2020]

Rebased onto master after #942's merge (registration-chain union only; the smart-extractor regions composed cleanly and the lane-affinity wiring is verified intact). Full gates green. Stays draft until #964 clears the shared smart-extractor overlap.

[gorkem2020]

Marking ready for review: both overlapping PRs (#942, #964) have merged and this branch is rebased onto current master with the full gate set green (typecheck, fresh dist, manifest verifier, full suite). The change is production-proven on our deployment: the same form has run live since 2026-08-04 with the uniform pipeline handling organic reflection bursts (create, merge, skip, and utility-veto classes all observed in one burst on day one).

[gorkem2020]

Rebased onto master after #972's merge (registration union only). Full gates green.

[rwmjhb]

Requesting changes at 1907ba6. The happy-path tests and required CI checks are green, but the new external-lane entry point has several data-loss and provenance regressions:

1. persistGatedCandidates creates a schema-incomplete synthetic admission audit at src/smart-extractor.ts:958 and forwards it into merge/support handling. withAdmissionAudit then replaces the target's complete admission_control record with only decision and reason. Please represent "already gated" separately and do not persist this synthetic audit.

2. The caller-provided reflection entry builder is consulted only by buildStoreEntry. Matched supersede, contextualize, and contradict paths construct auto-capture entries directly (src/smart-extractor.ts:2734, :2907, :2983), losing the reflection event/session provenance, heading, mapped kind, decay settings, and admission audit. Build these rows from the external entry and then layer the verdict-specific metadata onto it.

3. A dedupPrefilter failure at src/smart-extractor.ts:978 is retried through inline dedup; if the same search fails again, the outer catch only logs and queues no create. That silently drops an already-admitted reflection row. This path should fail open through the caller's builder.

4. flushPendingMerges drops every queued addition when the merge response is failed or malformed (src/smart-extractor.ts:2544). These mapped rows were previously direct-stored, so unresolved external-lane additions should fall back to create instead of disappearing.

The new tests cover CREATE/MERGE/SKIP happy paths and judge failure, but not these branches. Please add regressions for all four. Also cover same-burst duplicates: unlike the normal extraction route, persistGatedCandidates omits batchDedup, so identical rows in one burst can both be created.

[gorkem2020]

All five findings were correct, thank you for the careful read of the external-lane entry point. Fixed at 2e2f932, one point per finding:

1. Synthetic audit no longer exists. The pre-gated evaluation now carries the caller's own admission record, parsed from the built entry (preGatedFor(candidate)), so anything persisted downstream is the real gate audit, never a marker. The builder registration prebuilds each entry once and keeps {build, prebuilt, audit} together.
2. All verdict rows originate from the caller's entry. A shared externalVerdictEntry(candidate, overlay) bases supersede, contextualize, and contradict rows on the prebuilt entry (provenance, heading, mapped kind, decay, importance, audit intact) and layers only the verdict-specific fields (supersedes/fact_key/relations/contexts/state) on top. Extraction candidates keep the auto-capture shape unchanged.
3. Double search failure fails open. The processCandidate call-site catch now stores the caller-built row when the candidate is externally gated, so an admitted row can no longer be dropped by a failing dedup search.
4. Degraded batched merges fall back to create. flushPendingMerges takes the gated lane's createEntries and re-routes each external addition to a create when merge generation fails or apply throws; the extraction lane keeps its existing degrade semantics (it passes nothing).
5. Same-burst duplicates collapse. The gated lane now runs the same batchDedup as the extraction route, directly on the caller's vectors, with skipped accounting.

Five regression cells added, one per finding, in test/reflection-mapped-uniform-pipeline.test.mjs (review-round hardening block). Red-proofed: exactly those five fail against the pre-fix head 1907ba6 and all ten pass at 2e2f932. Full gates green (typecheck, fresh dist, manifest verifier, full suite).

[rwmjhb]

Thanks for the latest hardening pass. The earlier processing/merge exception paths now fail open, but two data-integrity gaps remain:

1. A matched row disappearing between dedup and mutation still silently drops the admitted mapped row. readMergeTarget treats a null getById result as a valid empty target; applyMergedContent then ignores both its own nullable getById and the nullable result from store.update, after which flushPendingMerges increments stats.merged and emits a persistence notification. The SUPPORT path similarly returns when its target is absent, while processCandidate still increments stats.supported. Please make null reads/updates fall back to the caller-built CREATE row for externally gated candidates, report success only after a durable mutation, and add regressions for vanished MERGE and SUPPORT targets (including a null update result).

2. The mapped-row audit parser does not match the production metadata shape. index.ts stores the gate record as a JSON string under metadata.admission_audit, but parseEntryAdmissionAudit reads only an object at metadata.admission_control. The new test masks this by inventing admission_control: { decision: "create", ... }; production emits an admission_audit string containing a pass_to_dedup record. Consequently MERGE and SUPPORT do not receive the actual mapped-row audit/provenance. Please parse the production field (including its nested JSON representation), or pass the original audit record directly, and replace the fixture with production-shaped metadata.

Non-blocking but worth correcting in the same area: the no-extractor fallback comment says a later burst will merge a duplicate stored during a search outage, but that path only performs the same pre-check and has no merge/reconciliation step.

The 10 focused uniform-pipeline tests pass and current GitHub CI is green. The local full-suite chain stalled in the unrelated existing update-consistency-lancedb.test.mjs; I am not treating that harness issue as an author blocker.

[gorkem2020]

Both findings were correct — the vanished-target class was real data loss and the audit parser was reading a field production never writes. Fixed at 961cbab:

1. Vanished targets now fail over to the caller-built row. readMergeTarget treats a null getById like an unreadable target (create fallback, no merge generated against empty content); applyMergedContent returns "updated" | "target-missing" — it bails before writing when the read comes back null AND when store.update itself returns null, and the persistence notification only fires after a durable update. flushPendingMerges and the single-call merge path route target-missing to the same fail-open used for degraded merges; handleSupport reports target-missing (null read or null update) and the support case falls back to a create instead of counting supported. The supersede handler's vanished-target fallback now also prefers the caller's prebuilt entry. Fallback rows for externally gated candidates are always the caller's own prebuilt entry (shape, provenance, audit intact).
2. The audit parser now reads the production field. parseEntryAdmissionAudit parses metadata.admission_audit including its nested JSON-string representation (the shape the mapped lane actually persists), and the fixtures were replaced with production-shaped metadata (full amac-v1 record with pass_to_dedup decision and mapped-lane provenance, serialized as the nested string). The masking admission_control fixture is gone.
3. The no-extractor fallback comment no longer claims a later burst merges the duplicate; that path only pre-checks and the comment now says so.

Four cells cover the round (vanished MERGE target at read, vanished during update via a null update result, vanished SUPPORT target, and the production-shape audit flow) — red-proofed: exactly those four fail at 2e2f932, all thirteen pass at 961cbab. Three store doubles in existing suites were brought up to the real update contract (MemoryEntry | null) since they previously returned undefined from successful updates. Full gates green (build, fresh dist, manifest verifier, full suite).

[rwmjhb]

Thanks for fixing the production audit parsing and vanished-target fail-open paths; those previous blockers are resolved. The uniform route still has three correctness gaps:

1. Cross-category MERGE can make the target disappear from both smart-category views. The dedup prompt explicitly allows cross-category verdicts, but applyMergedContent stamps metadata.memory_category from the incoming candidate while its store.update patch leaves the target's legacy category column unchanged. MemoryStore.list first SQL-prefilters on that column and then validates metadata.memory_category. For example, merging a patterns candidate into a preference row leaves category=preference and memory_category=patterns: a patterns list never fetches it, while a preferences list fetches and then rejects it. Please preserve the target category on merge, or update both representations atomically, and add a regression that queries the merged row through the real category-filtered list path.

2. The new same-burst guard drops rows before any category-aware or semantic decision. persistGatedCandidates applies batchDedup at cosine > 0.85 across all mapped rows, and batchDedup always discards the later item without considering category, mapped kind, exact text equality, or whether it adds information. The current test covers only identical pattern rows. A richer restatement or a related row from another reflection section can therefore be silently lost. Please restrict this shortcut to exact normalized duplicates, make it category-aware, or send non-identical pairs through the merge judge; add richer-row and cross-category regressions.

3. A fail-open gate marker can overwrite a target's complete audit. buildFailOpenResult produces only provenance/failedOpen/reason/error, but parseEntryAdmissionAudit casts it to AdmissionAuditRecord. MERGE/SUPPORT then replace the target's complete admission_control with that incomplete object, and admission statistics no longer see a decision. Please model fail-open evidence separately (or validate the union) and never replace a complete target audit with the marker.

Non-blocking follow-up: SUPPORT produces no reflection mdMirror entry, and MERGE is mirrored as generic smart-extraction, losing the reflection:<heading> provenance carried by the old path.

The 13 focused tests, full npm test, formal build/dist check, and GitHub CI all pass; none currently exercises these cases.

[gorkem2020]

All three were correct, fixed at c6c34e3:

1. A merge never reclassifies its target. applyMergedContent now stamps memory_category from the target's own metadata (column fallback) instead of the incoming candidate, so a cross-category merge can no longer desync the metadata from the legacy category column and drop the row out of both filtered views. The regression drives the merged row's column and patched metadata through the real filter functions (resolveCategoryFilterCandidates + matchesMemoryCategoryFilter, the exact pair list() uses for its SQL prefilter and post-validation) and asserts the row stays reachable in the target's view and invisible in the candidate's.
2. The same-burst guard collapses exact twins only. The vector-similarity shortcut is gone; the guard now drops a row only on exact normalized text equality within the same category. Richer restatements and same-text rows from different reflection sections proceed to the dedup judge. Regressions cover both (a richer restatement sharing an identical vector, and identical text under two categories).
3. A fail-open gate marker can no longer masquerade as an audit. parseEntryAdmissionAudit validates the parsed record (version: "amac-v1" plus a string decision) and returns nothing for {provenance, failedOpen, reason, error} markers, so MERGE/SUPPORT never overwrite a target's complete audit with one. Regression pins that no update patch carries a versionless audit object.

The non-blocking mirror note is also addressed for MERGE: batched merge jobs carrying an externally gated addition now notify with the caller's reflection:<heading> provenance (read from the prebuilt entry) instead of the generic extraction label, asserted in the category regression. SUPPORT remains notification-free on both lanes as before; happy to add a support-side notification as a follow-up if you want one.

Red-proofed: exactly the four new cells fail at 961cbab, all seventeen pass at c6c34e3. Full gates green (build, fresh dist, manifest verifier, full suite).

[rwmjhb]

Thanks for fixing the target-category merge, richer-restatement, and fail-open-audit cases. One same-burst data-integrity gap remains:

persistGatedCandidates builds the twin key from only candidate.category plus normalized text. Reflection lesson and decision rows both map to candidate category cases (and storage category fact), while retaining different mapped kinds, headings, importance, and decay policies. The reflection slicer emits Lessons before Decisions, so identical text under those two headings deterministically keeps the lesson and drops the durable decision before the dedup judge runs.

I reproduced this on c6c34e3 with one lesson and one decision carrying the same text: the result was created: 1, skipped: 1, and the only stored row was the lesson (importance=0.8, 7-day decay); the decision (importance=0.85, 45-day decay) was lost. The new "different reflection sections" test uses patterns versus preferences, so it verifies different categories but does not cover different mapped kinds sharing one category.

Please include the mapped kind or another caller-supplied lane identity in the exact-twin key, or otherwise collapse only exact duplicates from the same mapped lane, and add a lesson-versus-decision regression that proves neither row is silently discarded before semantic judging.

Also please replace the literal NUL delimiter currently embedded in src/smart-extractor.ts and dist/src/smart-extractor.js with a source escape or ordinary unambiguous delimiter. The actual 0x00 byte makes text tools such as rg classify these source files as binary.

Verification on the current head: the focused uniform-pipeline suite passes 17/17, npm run build succeeds and leaves generated files clean, the previously stalled LanceDB update test passes 6/6 standalone, and GitHub CI is green. The orchestrator's full-suite run itself timed out at 1200 seconds, so it did not verify the tests after that point.

[gorkem2020]

Both points fixed at 4259197:

1. The twin key now carries the caller's lane identity. The guard reads the prebuilt entry's mappedKind (reflection heading as fallback) and keys on [lane, category, normalized text], so a lesson and a decision with identical text — one candidate category, different kinds, importance, and decay — both reach the dedup judge. Your repro is the new regression: one lesson plus one decision with the same text now persist as two rows (created: 2, skipped: 0), with both mapped kinds asserted on the stored rows. Red-proofed: the cell reproduces created: 1, skipped: 1 against the previous head.
2. The literal NUL byte is gone. The delimiter was an editing accident (it also explains why some of our own text tooling had started classifying the file as binary — thanks for pinning it); the key is now built with JSON.stringify, no exotic delimiter at all. src/smart-extractor.ts and the rebuilt dist are clean: grep -rlaP '\x00' finds nothing outside historical README translations.

Full gates green (build, fresh dist, manifest verifier, focused suite 18/18, full suite exit 0).

[gorkem2020]

Rebased onto current master (ba9928f) after the #965 and #971 merges; only the test-chain registration needed a union, all gates re-run green (build, fresh dist, manifest verifier, full suite). Round-4 content unchanged.

[rwmjhb]

The latest head resolves the previous lesson-versus-decision lane collision and removes the literal NUL delimiter. One same-burst semantic-dedup gap still blocks the uniform route:

persistGatedCandidates now removes only exact normalized twins within a lane, then runs every survivor through dedupPrefilter independently. That prefilter searches only already-persisted store rows. Because all prefilters and verdicts are computed before this burst's CREATE entries are stored, related non-identical candidates in the same burst never see each other when the store has no pre-existing neighbor; both short-circuit to CREATE without a semantic judge call.

I reproduced this on 7cd18a4 with the focused test's short lesson and richer restatement, an empty store, and identical vectors. The result was created: 2, skipped: 0, llmCalls: [], with both texts persisted. The existing test named "keeps a richer same-burst restatement alive for the judge" uses the same empty-store setup and makeLlm({}); its assertion that two rows are stored proves only that the exact-twin guard did not drop either row, not that either reached the judge.

Please include eligible same-lane burst candidates in one another's semantic dedup context, then add a regression that explicitly observes a dedup-decision-batch call and verifies a MERGE/SKIP verdict prevents two unconditional CREATEs. Keep negative controls for unrelated rows and distinct mapped lanes so the fix does not reintroduce the lesson/decision loss.

Verification on the current head: the focused uniform-pipeline suite passes 18/18, the orchestrator full suite passes, npm run build succeeds with a clean generated tree, and current GitHub CI is green. Those tests do not currently exercise same-burst semantic adjudication.

[gorkem2020]

Fixed at cd4062b — the same-burst design is now complete end to end:

Same-lane siblings are part of each other's semantic dedup context. Earlier burst rows (same mapped kind and category, cosine at or above the store's own similarity threshold) join a candidate's neighbor list as virtual entries, so the empty-store short-circuit no longer bypasses the judge for related pairs; domain short-circuits like the preference-slot guard stay authoritative. A verdict against a sibling resolves after bulkStore assigns the sibling's real id and then reuses the normal machinery: SKIP drops the later row with accounting, MERGE queues into the sibling's stored row through the batched merge writer, SUPPORT updates its support stats, and anything unresolvable (sibling itself merged away, filtered by bulkStore, or an exotic verdict against a pending row) fails open to the caller-built create.

Your repro is the regression, both ways: the richer-restatement pair now produces one dedup-decision-batch call and, under a MERGE verdict, one stored row plus one merge onto it (created: 1, merged: 1); a SKIP-verdict variant pins created: 1, skipped: 1. Negative controls hold: dissimilar same-lane rows (below the similarity threshold) and the lesson-versus-decision cross-lane pair stay judge-free and persist independently — the harness LLM throws on any unexpected call, so those cells prove the absence of judge traffic, and the store double now registers bulk-stored rows for id lookups the way the real store does.

Red-proofed: exactly the two judge-observing cells fail against 7cd18a4, all twenty pass at cd4062b. Full gates green (typecheck, build, fresh dist, manifest verifier, full suite; added-line extraction count cross-checked, no control bytes).

[rwmjhb]

The same-burst semantic-neighbor design now fixes the previous adjudication gap, and the MERGE/SKIP plus lane-separation regressions are good. One fail-open blocker remains in the deferred sibling phase:

After the initial bulkStore, persistGatedCandidates iterates pendingSiblingVerdicts. The SUPPORT branch directly awaits handleSupport, but handleSupport can throw from either store.getById or store.update. That loop has no per-verdict exception guard, so an exception rejects the whole persistence call instead of adding the admitted candidate to followupCreates. It also abandons any follow-up creates/merges already accumulated and skips every later deferred verdict because those queues are flushed only after the loop.

I reproduced this on cd4062b with two same-lane siblings, a SUPPORT verdict, and a throwing store.update: dedup-decision-batch ran, the call rejected with Error: support write outage, and only the first sibling remained stored. The admitted second row was neither used as support nor fail-open created.

Please make each deferred verdict independently fail open: catch getById/update and other deferred-resolution exceptions, enqueue the caller-built row exactly once, continue processing later verdicts, and still flush previously accumulated follow-up work. Add regressions for throwing getById and throwing update, plus a multi-verdict case proving one failure does not discard earlier or later follow-ups.

Non-blocking hardening: storedIdForSurviving falls back to matching only text when bulkStore returns fewer rows; identical text is intentionally allowed across lanes, so include lane/category identity in that fallback to avoid binding a deferred verdict to the wrong stored row.

Verification on the current head: the focused uniform-pipeline suite passes 20/20, the orchestrator full suite passes, npm run build succeeds with a clean generated tree, and current GitHub CI is green. None of those tests currently injects a deferred SUPPORT exception.

[gorkem2020]

Both points addressed at c159242:

Deferred verdicts now degrade alone. Each entry in the deferred sibling loop resolves inside its own guard: a throwing getById/update (or any other resolution failure) logs, falls open to that candidate's caller-built row, and the loop continues, so follow-up merges queued by earlier verdicts still flush and later verdicts still resolve. The fallback enqueues exactly once: every push in the loop happens only after its awaits resolve, so a caught verdict has enqueued nothing yet, and the catch-path create is itself guarded so a double fault degrades to a logged drop rather than a rejection.

Lane-aware stored-row fallback. storedIdForSurviving's position-shift fallback now requires text plus category plus burst-lane identity (mappedKind, then _reflectionHeading, shared with the twin key through one laneFromMetadata helper), so a deferred verdict can no longer bind to another lane's row that merely shares the text.

Regressions (all red-first against the previous head): throwing getById; throwing update (your support write outage repro shape, whole call previously rejected with only the first sibling stored); a four-row multi-verdict cell proving one failing support neither discards the earlier queued merge (its content update still lands on the anchor row) nor skips the later support; and a cross-lane same-text cell where bulkStore drops the anchor and the fallback must refuse the other lane's row and fail open. Focused suite 24/24, full suite green, build clean, dist rebuilt.

[gorkem2020]

Weekly status note: this PR has been ready for re-review since August 10. All four review rounds were addressed, each within hours of the feedback, checks are green, and there are no open threads. The re-review request to @rwmjhb has been pending since then.

No rush intended, just keeping the thread current. Happy to address anything further whenever you have a moment.

[rwmjhb]

The deferred sibling SUPPORT fixes are effective: throwing reads/writes now fail open per verdict, earlier follow-ups still flush, later verdicts continue, and lane-aware stored-row resolution addresses the previous blocker. Two data-integrity issues remain:

1. Supersede invalidation can reject after a partial commit. persistGatedCandidates first commits the superseding rows with bulkStoreAndValidate, then unguardedly awaits applyPendingSupersedeInvalidations. A read/update exception while invalidating the old row propagates before deferred sibling work runs. I reproduced this on c159242: the new superseding row was stored, the old row remained active, and the call rejected with Error: invalidate outage. Please use an atomic supersede-and-store operation where available, or isolate each invalidation failure, continue later work, and downgrade the outcome to a plain CREATE rather than reporting it as superseded. Add throwing and null-returning invalidation regressions with later deferred work.

2. MERGE and SUPPORT discard mutation-specific fail-open admission evidence. The production fail-open marker intentionally lacks version/decision, so parseEntryAdmissionAudit returns undefined; those verdicts do not store the caller-built entry carrying metadata.admission_audit. In a merge reproduction, the resulting target metadata had no admission_audit, no new admission_control, and no failedOpen marker, even though unevaluated content became durable. Please model complete audits and fail-open evidence as a validated union or separate append-only field, preserving an existing complete target audit while recording that this mutation bypassed evaluation. Cover production-shaped MERGE and SUPPORT markers.

Non-blocking hardening: when CONTEXTUALIZE/CONTRADICT targets vanish, fall back to an ordinary CREATE without a relation to a nonexistent row.

Verification on the current head: the focused uniform-pipeline suite passes 24/24, the orchestrator full suite passes, npm run build succeeds with a clean generated tree, and current GitHub CI is green. These failure and provenance cases are not covered.

[gorkem2020]

All three items addressed in 7278eeb:

1. Supersede invalidation is now isolated from the committed replacement. Each invalidation failure, thrown or an update that writes nothing, is caught per row; later invalidations and the deferred sibling verdicts continue, and the outcome downgrades to a plain CREATE, stripping the supersedes claim and relation from the already-committed replacement so the pair never reports a supersede that did not happen.

2. Fail-open admission markers now survive as evidence. Parsing distinguishes complete audits from the production marker shape: complete audits keep replacing admission_control as before, while markers never touch an existing audit and are appended to a new append-only admission_bypass_events field (capped at the newest 20), so a MERGE or SUPPORT that made unevaluated content durable stays visible on the target.

3. The non-blocking hardening is included: CONTEXTUALIZE and CONTRADICT verify the target still exists and fall back to an ordinary create without a relation when it vanished.

Regression coverage: throwing and nothing-written invalidations with later deferred sibling work continuing, plus production-shaped MERGE and SUPPORT fail-open markers asserting audit preservation and bypass evidence. All four fail on the pre-fix source and pass after; the focused suite is 28/28, the full chain and build are green, and the dist is rebuilt in the same commit.

[rwmjhb]

Thanks for addressing the previous round. I re-reviewed head 7278eeb: the focused reflection suite passes 28/28, the full suite and GitHub checks are green, and npm run build leaves the generated tree clean. The main isolation and admission-bypass fixes are directionally right, but there are still data-integrity blockers in the failure paths.

1. Deferred supersede invalidation can bind the old row to an unrelated create after bulkStore filters an earlier entry. handleSupersede records a raw entryIndex from createEntries, while applyPendingSupersedeInvalidations later indexes the possibly shorter returned array (src/smart-extractor.ts:3268-3277, :3307). bulkStoreAndValidate explicitly accepts a shorter result. I reproduced a three-row batch where row 1 was filtered, row 2 was the surviving replacement, and row 3 was an unrelated create: the old row's superseded_by became new-2 (row 3) instead of new-1 (the replacement). Resolve pending invalidations by stable entry identity, as the sibling-verdict path already does when positions shift, and do not invalidate unless the exact replacement is found.

2. Grouped merges preserve admission evidence only from the first addition. Jobs group every candidate sharing a matchId, but flushPendingMerges passes only job.additions[0]?.admissionAudit at src/smart-extractor.ts:3009. A two-addition probe with a fail-open marker on the second row merged both texts but produced no admission_bypass_events entry for that row. Aggregate every addition's complete audit/fail-open evidence into the capped append-only field.

3. The target-disappearance hardening is still incomplete. handleContextualize now performs an unguarded getById at src/smart-extractor.ts:3464; in the ordinary extraction loop the exception is only logged, so the admitted candidate is dropped instead of falling back to an unlinked CREATE. Separately, handleContradict ignores a null result from the target update at :3545 and still derives contradicts from the earlier read; a null-update repro returned a new row with a dangling relation. Treat read/update failure or null as target-missing and persist the candidate without a relation.

4. A failed supersede is not fully reclassified as CREATE. downgradeSupersedeToCreate ignores a null cleanup result at src/smart-extractor.ts:3388, so a second write failure can leave the replacement's supersedes claim durable while the old row remains active, despite the success-style downgrade log. Also, stats.superseded is incremented before deferred invalidation at :2339; with one failed and one successful invalidation my repro returned 2 instead of 1. Check the cleanup result, surface an unresolved repair state when stripping cannot be confirmed, and count only confirmed invalidations.

Please add regressions for the filtered-array identity shift, multi-addition bypass evidence, contextualize read failure, contradict null update, cleanup null, and confirmed-only supersede statistics.

[gorkem2020]

All four addressed at c6ca9f9:

1. Identity-bound deferred invalidation. PendingSupersedeInvalidation now carries the queued entry object. applyPendingSupersedeInvalidations trusts positions only when bulkStore returned the full batch; otherwise it binds by stable identity (text + category + lane, first unclaimed, the same fallback the sibling-verdict resolver uses) and skips entirely when the exact replacement is absent. Your three-row repro is the first new regression: superseded_by lands on the true replacement, never the row that shifted into its position, and a filtered replacement leaves the old row untouched.

2. Grouped-merge evidence. flushPendingMerges now passes every addition's evidence. The first addition keeps the existing single-evidence semantics, and every later addition's audit or fail-open marker is appended into the capped admission_bypass_events field via a new appendAdditionalAdmissionEvidence helper. Regression: a two-addition grouped merge with the marker on the second row records it.

3. Contextualize/contradict fallbacks. The contextualize target read is guarded: a throwing read logs the failure and stores the candidate as an unlinked create instead of dropping it through the generic processing-failure catch. Contradict persists its contradicts relation only when the evidence update confirms a write; a null update (or a thrown read/update) logs target-missing and stores the candidate unlinked. Both regressions assert the landed row carries no relation.

4. Confirmed-only downgrade and statistics. downgradeSupersedeToCreate checks the strip write's result and, when the strip cannot be confirmed, logs an explicit UNRESOLVED supersede repair state instead of the success-style downgrade line. stats.superseded counts only confirmed invalidations: the immediate path counts on invalidation success and the deferred path counts inside applyPendingSupersedeInvalidations. The regression reproduces your two-supersede case (one failed invalidation) and asserts a count of 1.

Six new regressions cover the requested scenarios (filtered-array identity shift in both directions, multi-addition bypass evidence, contextualize read failure, contradict null update, unresolved cleanup, confirmed-only statistics), red-proofed against the previous head: with the source change stashed, exactly those six fail and the prior 28 stay green. Focused file 34/34, full suite green, npm run build output committed.

[rwmjhb]

Thanks for the comprehensive round. On head c6ca9f9, all six requested regressions are fixed: the focused suite passes 34/34, the full suite and GitHub checks are green, and the prior supersede/evidence/target-fallback blockers no longer reproduce. One combined partial-batch case still breaks the PR's core dedup guarantee:

1. Multiple deferred verdicts cannot reuse one surviving sibling anchor after bulkStore filters an unrelated row. In the shortened-result branch of storedIdForSurviving (src/smart-extractor.ts:1298-1328), the first lookup finds the anchor and adds its ID to claimedIds. A later MERGE or SUPPORT with the same siblingIndex then excludes that same ID, resolves no target, and fails open to CREATE. The full-length branch correctly reuses the positional ID, so behavior changes only because an unrelated create was filtered. I reproduced this with one filtered row, one surviving anchor, and two SUPPORT verdicts: expected stats.supported === 2, actual was 1, with the second candidate falling back to a duplicate create. Cache the resolved survivingIndex -> stored ID mapping; use claimedIds only while assigning distinct surviving entries, then let every verdict for that index reuse the cached ID.

2. Complete successful audits are being stored in the bypass-only history. appendAdditionalAdmissionEvidence at src/smart-extractor.ts:3909-3924 appends every later AdmissionWriteEvidence, including normal amac-v1 pass audits, to admission_bypass_events. That mislabels evaluated mutations and lets normal grouped merges consume the 20-record cap and evict genuine fail-open evidence. Keep admission_bypass_events exclusive to fail-open markers and use a separately named typed history for additional complete audits.

Please add the combined filtered-result/shared-anchor regression. Non-blocking cleanup: interpretDedupVerdict already declares context_label, so the redundant (data as any) casts can be removed.

[gorkem2020]

Both findings fixed on head dc49f36.

1. storedIdForSurviving now caches each surviving index's resolved ID, so every later verdict for that index reuses the same row; claimedIds participates only in the identity fallback that separates distinct surviving entries. The requested combined regression is in (one filtered unrelated row, one surviving anchor, two SUPPORT verdicts): asserts stats.supported === 2, no duplicate create, and both support writes bound to the same row.

2. appendAdditionalAdmissionEvidence now routes evidence by kind: admission_bypass_events stays exclusive to fail-open markers, and additional complete audits append to a new capped admission_control_history field. A second regression pins the routing (complete audit lands only in the history, the fail-open marker alone occupies the bypass field, and the target's own admission_control stays untouched).

Also removed the redundant (data as any) casts in interpretDedupVerdict.

Both new tests were red-proofed against the unfixed source (each fails exactly as the review describes: supported 1 instead of 2 with a duplicate create, and a pass audit occupying the bypass history). Focused suite 36/36; full suite and typecheck green.

[rwmjhb]

Thanks for addressing the previous round. On head dc49f36, the cached shared-anchor mapping and split audit histories fix the two prior blockers. Two same-burst paths still break the uniform dedup behavior:

1. Deferred verdict chains do not resolve transitively. createSlotBySurviving is populated only when a survivor directly queues a CREATE. If B merges into A, B remains a virtual sibling but has no create slot; when C then merges into B, storedIdForSurviving(B) returns undefined and C falls open to CREATE. I reproduced this in an empty store with B MERGE A and C MERGE B: the result was created: 2, merged: 1, with both A and C persisted despite both semantic merge verdicts. Please track each survivor's final durable anchor transitively, including chains that end at an existing row, and add regressions for A-created/B-merges-A/C-merges-B plus an anchor that itself merges into a stored row.

2. The preference-slot short circuit suppresses eligible sibling adjudication. persistGatedCandidates yields to burst siblings only for the exact No similar memories found short circuit. When a stored preference is a different item from the same brand, the preference guard returns CREATE and related same-lane siblings never reach the batch judge. With an existing fries from McDonald's row and two reworded burger candidates, both burger rows were created and the dedup judge was never called. When the guard excludes stored neighbors but eligible burst siblings exist, please adjudicate those siblings alone and add this same-brand/different-stored-item/same-incoming-item regression.

Current split GitHub CI checks are green. The local serial npm test run hit the orchestrator's 180-second harness limit after all visible tests had passed, so I am not treating that timeout as an assertion failure. The two behavior reproductions above are the blockers.

[gorkem2020]

Both round-8 blockers fixed on head 8276926.

1. Deferred verdicts now resolve through a transitive durable-anchor map: a survivor that merged/supported/skipped into an earlier sibling records that sibling as its anchor, and one whose verdict targeted an existing stored row records the row id. storedIdForSurviving chases the chain (indices strictly decrease, so it terminates) before falling open. Your B-merges-A / C-merges-B reproduction now lands as created: 1, merged: 2 with one grouped update on A's row, and a chain ending at a stored row resolves there. Both requested regressions are in.

2. A create-decision short circuit (the empty-store bypass and the preference-slot guard) now yields to eligible burst siblings instead of suppressing adjudication: the siblings are judged alone, since the short circuit carries no stored rows, and non-create short-circuits stay authoritative. The same-brand/different-stored-item/same-incoming-item regression is in: with a stored different-item preference and two rewordings of one new item, the judge runs once and only the first wording persists.

All three new tests were red-proofed against the unfixed source (the chain case reproduces exactly your created: 2, merged: 1; the guard case shows the double create with the judge never called). Focused suite 39/39; full suite, typecheck, and build green.

[rwmjhb]

Thanks for fixing the previous transitive-MERGE and preference-slot cases. On head 8276926, the 39 focused tests and full suite pass, npm run build succeeds, and rebuilding leaves the worktree clean. One sibling-chain variant still breaks the durable-anchor guarantee:

A SKIP anchor is discarded before a later sibling can resolve through it. interpretDedupVerdict only includes matchId for merge/support/contextualize/contradict/supersede, not for skip (src/smart-extractor.ts:2720-2724). The new bookkeeping records anchorSiblingBySurviving for a SKIP only when pre.matchId is present (:1222-1231). As a result, when B is judged SKIP against A and C is judged MERGE against B, B has no recorded path to A; deferred resolution treats B as having no persisted row and fail-opens C to CREATE.

I reproduced this with three same-lane paraphrases in an empty store. The batch judge returned B SKIP A and C MERGE B; the result was created: 2, merged: 0, skipped: 1, with both A and C durable and no merge-writer call. The existing regression covers B MERGE A -> C MERGE B, so it does not exercise this path.

Please preserve the valid sibling target for SKIP verdicts (or otherwise record the skipped survivor's durable anchor), and add regressions for B SKIP A -> C MERGE B and the analogous SUPPORT-through-skipped-anchor case.

[gorkem2020]

Fixed on head shown below: interpretDedupVerdict now carries the skip verdict's target, so a same-burst SKIP records its sibling anchor (and a stored-row SKIP records its row anchor) while the skip handler itself keeps ignoring the target. Your reproduction resolves as created: 1, merged: 1, skipped: 1 with the merge landing on A's row, and the SUPPORT-through-skipped-anchor case resolves the same way.

Both requested regressions are in and were red-proofed against the unfixed source (each reproduces the fail-open duplicate exactly as described). Focused suite 41/41; full suite, typecheck, and build green with a clean worktree after rebuild.

[rwmjhb]

Re-reviewed head a0d556b (functional fix at d7a10c9). The previous SKIP-anchor blocker is resolved: the focused suite now includes both B SKIP A -> C MERGE B and B SKIP A -> C SUPPORT B, and all 41 mapped-row pipeline regressions pass. The full test suite passes, npm run build succeeds, generated output remains clean, and current required GitHub checks are green.

I found no remaining merge blocker. The fail-open behavior during sustained dedup-judge outages and the added serial LLM work inside the reflection lock remain operational tradeoffs, but they are bounded and do not invalidate this fix.