Follow-ups for SMST deferred hashing + copy-on-write historical proofs [#1432]#1447
Merged
Merged
Conversation
…ts, up to 3.4x faster ingest insertAt re-hashed every node on the root->leaf path on each insert, so a shared upper node was re-hashed once per descendant insert. Record structure and counts on insert (node.hash = nil) and fill hashes lazily once when the root is needed (GetRoot / snapshot proof / rebuild) via ensureHashed, hashing each node once per flush instead of once per insert that passes through it. Roots and proofs are byte-identical to per-insert hashing (same Merkle function, different hashing order): a 200k-insert fingerprint over flush roots and sampled proofs matches the previous implementation exactly, and per-N roots match at 100k / 500k / 1M / 10M. Hashing mutates nodes, so live-tree read paths that may fill hashes take the write lock; the live-tip proof path keeps a read-lock fast path once the hashes are already filled. Adds TestDeferredHashFingerprint (root/proof determinism guard), live-tip and historical-rebuild proof-path tests, and an env-gated build profile (TestSMSTBuildProfile) reproducing the ingest/RAM numbers.
… drop the rebuild-DoS limiter Historical snapshot proofs rebuilt the whole tree from the artifact log on every request (O(N) SHA-256), which a validator could turn into a rebuild-DoS by cycling distinct counts — guarded until now by a per-validator distinct-count quota. This makes the live tree copy-on-write and serves committed-count proofs from retained snapshots in O(depth). - insertCOW rewrites only the root->leaf path and shares untouched subtrees, so a captured root stays valid. Roots and proofs are byte-identical to Insert (same Merkle function, hashing deferred identically); the on-chain commitment is unchanged. - The store captures a retained snapshot at every committed (flush) count and serves its proofs from shared nodes in O(depth). On restart, recover() re-captures a snapshot at each committed count in one copy-on-write replay, so historical proofs stay O(depth) after a restart. - A proof at a non-committed count is rejected outright (its root can never match the on-chain commitment), so a proof request can no longer trigger a rebuild. The rebuild path remains only as a defensive cold-start fallback. - With the rebuild-DoS surface gone, the per-validator distinct-snapshot-count limiter is removed. Byte-identical roots/proofs verified across the suite; live, retained-historical and post-restart proof paths plus the concurrent-readers test are green under -race.
…essions Byte-identity of eager Insert, copy-on-write insert, and rebuild-from-log across roots and raw proofs at many N (through depth-expansion boundaries); served transport proofs verify against the eager root. Plus two regression tests for the load-test findings this PR addresses: a non-committed proof count is rejected outright (no O(N) rebuild), and an early committed root is served from a re-captured retained snapshot in O(depth) after a restart, byte-identical to the pre-restart root.
…jsonl Recover used distributionHistory counts as the only durable flush markers, so a warn-only dist append failure made an on-chain flush look non-committed after restart. Write count+root to flushed_roots.jsonl on flush, union it on recover, backfill for pre-roots stores, and test surviving a wiped distributions file.
…ush profiles SMST_DEFERRED_HASH and SMST_COW default on; set either to 0 for profiling baselines. When COW is off, pin flushed counts via the process snapshot cache (WarmSnapshot/PrebuildSnapshot) so early 1/3 commits stay cached without retained trees. Add a 30-flush early-snapshot profile and coverage for defaults and COW-off early cache behavior.
… toggle When COW is off, PrebuildSnapshot can deep-clone under the write lock (default) or rebuild from artifacts without holding the lock (SMST_SNAPSHOT_IN_MEMORY_CLONE=0, v0.2.14-style). Refresh follow-up PR notes and Flush30 profile matrix.
Fan out deferred hash fill across GOMAXPROCS (default on); eager path hashing stays serial. Add root-equality coverage and deferred×parallel profile matrix; refresh follow-up PR notes.
Collaborator
|
/run-integration |
x0152
approved these changes
Jul 15, 2026
d-bogdan-engenious
pushed a commit
to d-bogdan-engenious/gonka
that referenced
this pull request
Jul 16, 2026
x0152
pushed a commit
to x0152/gonka
that referenced
this pull request
Jul 17, 2026
gonka-ai#1432] (gonka-ai#1447) * perf(poc/artifacts): defer SMST hashing to flush — byte-identical roots, up to 3.4x faster ingest insertAt re-hashed every node on the root->leaf path on each insert, so a shared upper node was re-hashed once per descendant insert. Record structure and counts on insert (node.hash = nil) and fill hashes lazily once when the root is needed (GetRoot / snapshot proof / rebuild) via ensureHashed, hashing each node once per flush instead of once per insert that passes through it. Roots and proofs are byte-identical to per-insert hashing (same Merkle function, different hashing order): a 200k-insert fingerprint over flush roots and sampled proofs matches the previous implementation exactly, and per-N roots match at 100k / 500k / 1M / 10M. Hashing mutates nodes, so live-tree read paths that may fill hashes take the write lock; the live-tip proof path keeps a read-lock fast path once the hashes are already filled. Adds TestDeferredHashFingerprint (root/proof determinism guard), live-tip and historical-rebuild proof-path tests, and an env-gated build profile (TestSMSTBuildProfile) reproducing the ingest/RAM numbers. * perf(poc/artifacts): copy-on-write SMST — O(depth) historical proofs, drop the rebuild-DoS limiter Historical snapshot proofs rebuilt the whole tree from the artifact log on every request (O(N) SHA-256), which a validator could turn into a rebuild-DoS by cycling distinct counts — guarded until now by a per-validator distinct-count quota. This makes the live tree copy-on-write and serves committed-count proofs from retained snapshots in O(depth). - insertCOW rewrites only the root->leaf path and shares untouched subtrees, so a captured root stays valid. Roots and proofs are byte-identical to Insert (same Merkle function, hashing deferred identically); the on-chain commitment is unchanged. - The store captures a retained snapshot at every committed (flush) count and serves its proofs from shared nodes in O(depth). On restart, recover() re-captures a snapshot at each committed count in one copy-on-write replay, so historical proofs stay O(depth) after a restart. - A proof at a non-committed count is rejected outright (its root can never match the on-chain commitment), so a proof request can no longer trigger a rebuild. The rebuild path remains only as a defensive cold-start fallback. - With the rebuild-DoS surface gone, the per-validator distinct-snapshot-count limiter is removed. Byte-identical roots/proofs verified across the suite; live, retained-historical and post-restart proof paths plus the concurrent-readers test are green under -race. * test(poc/artifacts): differential Insert==COW==rebuild + finding regressions Byte-identity of eager Insert, copy-on-write insert, and rebuild-from-log across roots and raw proofs at many N (through depth-expansion boundaries); served transport proofs verify against the eager root. Plus two regression tests for the load-test findings this PR addresses: a non-committed proof count is rejected outright (no O(N) rebuild), and an early committed root is served from a re-captured retained snapshot in O(depth) after a restart, byte-identical to the pre-restart root. * fix(poc/artifacts): release write lock before retained proof I/O * fix(poc/artifacts): persist flush roots independent of distributions.jsonl Recover used distributionHistory counts as the only durable flush markers, so a warn-only dist append failure made an on-chain flush look non-committed after restart. Write count+root to flushed_roots.jsonl on flush, union it on recover, backfill for pre-roots stores, and test surviving a wiped distributions file. * fix(poc/artifacts): fill live-tip hashes under Lock, serve proofs under RLock * feat(poc/artifacts): env toggles for deferred hashing and COW + 30-flush profiles SMST_DEFERRED_HASH and SMST_COW default on; set either to 0 for profiling baselines. When COW is off, pin flushed counts via the process snapshot cache (WarmSnapshot/PrebuildSnapshot) so early 1/3 commits stay cached without retained trees. Add a 30-flush early-snapshot profile and coverage for defaults and COW-off early cache behavior. * feat(poc/artifacts): tip Prebuild in-memory clone vs artifact rebuild toggle When COW is off, PrebuildSnapshot can deep-clone under the write lock (default) or rebuild from artifacts without holding the lock (SMST_SNAPSHOT_IN_MEMORY_CLONE=0, v0.2.14-style). Refresh follow-up PR notes and Flush30 profile matrix. * feat(poc/artifacts): multicore ensureHashed behind SMST_PARALLEL_HASH Fan out deferred hash fill across GOMAXPROCS (default on); eager path hashing stays serial. Add root-equality coverage and deferred×parallel profile matrix; refresh follow-up PR notes. --------- Co-authored-by: kAIPraxisBot <272124397+kAIPraxisBot@users.noreply.github.com> Co-authored-by: akup <ak@neonavigation.com> Co-authored-by: DimaOrekhovPS <dima.orekhov@productscience.ai>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Base:
feat/smst-tree-cache(or merge into #1432)Branch:
ak/smst-tree-cache-followupsSummary
Review follow-ups on top of PR #1432 (SMST deferred hashing + copy-on-write historical proofs). No consensus / root-byte changes; same Merkle semantics.
Fixes
Retained historical proofs no longer hold the write lock across artifact I/O
After taking a retained COW view, unlock and re-take
RLockso ingest and concurrent proofs are not serialized behindgetArtifactByNonce.Durable flush boundaries independent of
distributions.jsonlPersist
count+ root toflushed_roots.jsonlon flush. Recover unions that journal with distribution history so a warn-only dist-append failure cannot make an on-chain flush look non-committed after restart. Backfill for stores that only had dist history.Live-tip deferred hash fill under
Lock, proofs underRLockensureHashedruns under write lock only, then retries onto the read-lock fast path.getRoot/GetRootAt/GetFlushedRootpreferRLockwhen already hashed.Profiling / toggles (defaults unchanged for production)
0/falseSMST_DEFERRED_HASHSMST_COWInsert(no path-copy retain at flush)SMST_SNAPSHOT_IN_MEMORY_CLONEPrebuildSnapshotrebuilds from artifacts into the process cache without holding the write lock (v0.2.14Warm/Prebuildpath)SMST_PARALLEL_HASHensureHashed; default fans out deferred fill acrossGOMAXPROCSTip
PrebuildSnapshotwhen count equals the live tip:SMST_SNAPSHOT_IN_MEMORY_CLONE=1retainedSMST_SNAPSHOT_IN_MEMORY_CLONE=0Cold path (tip already past, nothing retained, committed count) still rebuilds into the cache.
SMST_PARALLEL_HASHaccelerates deferredensureHashed(independent subtrees, min 256 leaves). Eager per-insert path hashing stays serial (parent depends on child).Profiles:
TestSMSTBuildProfile(insert vsGetRoot) andTestSMSTStoreFlush30Profile(N leaves, 30 flushes, snap at flush #10).Measured — illustrative (N=300k)
Build profile (insert all leaves, then one full
GetRoot):Flush30 (COW on; each flush only fills hashes dirtied since last flush):
Multicore helps the big deferred fill (~6× on full-tree
GetRoot). Eager is unchanged. Incremental flushes already hash little, so parallel vs serial there is close.Snapshot modes (deferred, COW off, snap at 1/3):
Commits
fix(poc/artifacts): release write lock before retained proof I/Ofix(poc/artifacts): persist flush roots independent of distributions.jsonlfix(poc/artifacts): fill live-tip hashes under Lock, serve proofs under RLockfeat(poc/artifacts): env toggles for deferred hashing and COW + 30-flush profilesfeat(poc/artifacts): tip Prebuild in-memory clone vs artifact rebuild togglefeat(poc/artifacts): multicore ensureHashed behind SMST_PARALLEL_HASHTest plan
go test ./poc/artifacts/ -count=1go test ./poc/artifacts/ -race -run 'TestCOW|TestDeferred|TestFlushedRoots|TestSMSTCOW|TestSMSTDefaults|TestParallelHash' -count=1TestSMSTDefaultsDeferredAndCOW)Notes for #1432 authors / reviewers
These commits are intended to land on top of #1432 (or be folded into that PR).