Upgrade v0.2.14#1267
Conversation
Baseline measurements for duplicate attack mitigation work: - Insert throughput: 2M ops/sec in-memory, 330K with flush - Proof generation: 4.8M ops/sec - Proof size at 5M artifacts: ~900 bytes (28 hashes) - Recovery time at 1M artifacts: 742ms
- Create interface.go with ArtifactStore interface - Rename store.go to mmr_store.go - Rename ArtifactStore struct to MMRArtifactStore - Update ManagedArtifactStore to use interface type
Sparse Merkle Sum Tree where nonce determines tree position: - Duplicates impossible by design (same nonce = same slot) - Sum property enables dense index navigation - Proof includes sibling counts for robust verification - Dynamic depth expansion (default 24, max 32)
Compare MMR vs SMST: - Insert: 2M vs 355K ops/sec (5.6x slower) - Recovery: 742ms vs 3037ms at 1M artifacts (4x slower) - Proof size: Variable vs fixed 864 bytes (SMST wins at scale) Trade-off is acceptable given duplicate prevention is required.
Add use_smst field (field 13) to PocParams: - Chain: params.proto + regenerated params.pb.go - Testermint: AppExport.kt PocParams data class Default true for new deployments. Integration points documented for future ManagedArtifactStore factory selection.
Initial scaffold so the v0.2.13 branch has a registered handler before
any feature PRs land. Modeled on v0.2.9's clean shape rather than
v0.2.12's loaded one — features will be added as PRs merge.
What's here:
- app/upgrades/v0_2_13/constants.go: UpgradeName = "v0.2.13"
- app/upgrades/v0_2_13/upgrades.go: CreateUpgradeHandler with the
capability-version fix + RunMigrations. Migration steps land
between those two lines as features merge. Header comment spells
out the conventions (small functions, return errors, ConsensusVersion
bump if state-breaking, keeper additions threaded through).
- app/upgrades/v0_2_13/upgrades_test.go: TestUpgradeName pins the
constant so a typo can't slip past CI.
- app/upgrades.go: registered v0_2_13.CreateUpgradeHandler with the
minimal keeper signature. Add keepers to both call sites as
individual migrations need them.
InferenceModule.ConsensusVersion stays at 14 (no state-breaking changes
yet). The first PR that introduces one should bump it AND add the
matching RegisterMigration in registerMigrations() in app/upgrades.go.
Build + all upgrade-handler tests pass.
* fix: dynamic per-msg gasWanted in DAPI; bypass HardwareDiff and ClaimRewards DAPI was setting every batch's gasWanted to a constant BatchGasLimit (1B after #1120). Since Cosmos charges fees on gasWanted (not gasUsed), every non-bypass tx was billed at the worst-case PoC commit ceiling regardless of what it actually consumed. With min_gas_price re-enabled at 10 ngonka, that pattern would burn ~1000 GNK/host/year just on routine bookkeeping (see /tmp/gonka-gas-analysis/report.md for the 24h sample). Two changes: 1. DAPI sizes gasWanted per-batch. gas_estimate.go: per-message-type lookup table sized at observed p99 × ~1.5 from a 24-hour mainnet sample (~385K txs). The two known linear-scaling messages mirror their on-chain ConsumeGas formula: - MsgPoCV2StoreCommit: base + sum(entry.Count) × per_count - MsgMLNodeWeightDistribution: base + total_node_entries × per_node tx_manager.go: getSignedBytes takes gasWanted instead of using the constant. broadcastMessage / BroadcastMessages route through internal *AtAttempt helpers that pass the retry attempt count to estimateBatchGas, which doubles the estimate per attempt. The retry path in sendTxs uses tx.TxInfo.Attempts so OOG-triggered retries actually escape the loop instead of repeating the same too-low gas. errors.go: "out of gas" added to retryablePatterns so under-estimated txs requeue rather than failing permanently. 2. Chain bypasses two more routine messages. ante_fee.go: MsgSubmitHardwareDiff and MsgClaimRewards added to isExemptMessageType. Both are protocol-driven host duties on a fixed schedule (HardwareDiff fires per-block on actual hardware change, ClaimRewards is once per epoch per host). Neither is a sybil-attack vector the way MsgPoCV2StoreCommit is — that one keeps its count-proportional fee as the actual sybil defense. With these added to bypass, the only remaining fee-paying routine messages are MsgPoCV2StoreCommit (intentional) and MsgSubmitSeed (~7/host/day, ~2 GNK/host/year at min_gas_price=10). Combined effect on the 100 GNK/year budget at min_gas_price=10: - Before: ~1051 GNK/host/year (Option A sizing) — way over budget - After: ~24 GNK/host/year (sample size 30 hosts) — comfortable margin Initial table values come from /tmp/gonka-gas-analysis/report.md. Plan is to re-pull a fresh sample after some weeks of mainnet activity and tighten the numbers; the table is a single switch statement so it's trivial to re-tune. * test: cover gas_estimate.go and applyGasAndFee Three groups of tests: 1. Per-msg-type estimates pinned. TestEstimateMsgGas_KnownTypes asserts each switch case returns its declared constant. If someone retunes a value (e.g. after re-sampling mainnet) the test fails so the change shows up in the diff explicitly. 2. Linear-scaling formulas verified. The two formula-based estimates (PoCV2StoreCommit, MLNodeWeightDistribution) are tested at multiple payload sizes to confirm they grow linearly with the on-chain ConsumeGas formula they mirror. 3. Linter-style guard against forgotten message types. estimateMsgGas was refactored to expose an internal lookupMsgGas that returns (estimate, explicit) so a test can iterate InferenceOperationKeyPerms (the warm key's full authz permission list) and assert each msg type has an explicit case in the switch. Several legitimate per-type estimates happen to coincide with gasDefaultEstimate (500K), so a value comparison wouldn't work. Also extracted applyGasAndFee from getSignedBytes so the gas/fee math is unit-testable without keyring + signing setup. New tests cover: - zero min_gas_price produces zero fees regardless of gasWanted (the v0.2.12 mainnet config) - non-zero min_gas_price produces fee = gasWanted × minGasPrice - gasWanted of 0 or above BatchGasLimit clamps to BatchGasLimit Tests also cover estimateBatchGas: tx-overhead addition, retry-attempt multiplier, BatchGasLimit cap, empty-batch safe path. * fix: review feedback on fix-fees-again Three changes from PR review: 1. SubmitHardwareDiff no-op idempotency check (Should fix). The handler now skips the SetHardwareNodes write when the resolved node set is element-wise identical to what's already stored. Without this guard, MsgSubmitHardwareDiff was a free spam vector after being added to NetworkDutyFeeBypassDecorator: a participant could submit the same diff every block at zero cost, with only the bypass GasCap limiting them. The check is also a fix for the "DAPI sends diffs every ~11 minutes even when nothing changed" behavior the PR body already noted. Cosmos has no rate limit primitive at the msg-handler layer, so an idempotency check is the right shape: it's local to the handler, provably safe (output state identical to the previous write), and doesn't introduce timing-sensitive logic. Both lists are sorted by LocalId at this point — the existing handler sorts updatedNodes before write, and prior writes were sorted the same way — so a length + pairwise proto.Equal compare is sufficient. 2. PoCV2 / txOverhead constant comments call out the FeeParams dependency and authz-wrap absorption explicitly (Nits). gasPoCV2Base and gasPoCV2PerCount mirror governance-tunable FeeParams.{base_validation_gas, gas_per_poc_count}. If governance bumps either, the estimator silently underestimates. The OOG retry path absorbs this gracefully but at the cost of extra block time, so the constants need a re-tune in the same governance window. Comment spells this out and points at the future work (read FeeParams at startup) as a cleaner long-term solution. txOverheadGas comment now mentions the authz MsgExec wrapper as a contributor — mainnet hosts almost always run in authz mode (warm key signs, cold pays), so anyone retuning should keep this in mind. 3. Empty-batch test for BroadcastMessages (Nit). TestBroadcastMessages_EmptyBatch_NoOp pins the early-return guard at the top of broadcastMessagesAtAttempt so a future refactor can't inadvertently route a zero-msg batch into BuildUnsignedTx, which produces a confusing chain-side decode error far from the cause. Tests added: - TestHardwareNodesUnchanged: unit test for the helper - TestMsgServer_SubmitHardwareDiff_IdempotentOnNoChange: end-to-end confirms identical re-submission and empty diff both succeed without state divergence - TestBroadcastMessages_EmptyBatch_NoOp: covers all three empty-msgs paths (public BroadcastMessages, internal helper with non-zero attempt, and explicit empty slice) Skipped from review: the suggestion to read FeeParams at DAPI startup and use those values dynamically. That's a larger change with a runtime dependency — the comment now flags it as future work. Easier near-term fix is to re-tune the constants when governance changes the params. * fix: address patimen review feedback Six items from PR #1129 review: 1. HardwareDiff no-op compares operational fields only. The previous proto.Equal compared every field, including Version which is explicitly informational per hardware_node.proto. An attacker (or buggy DAPI) could flap version to defeat the no-op check. New helper hardwareNodeOperationalEqual compares only the load-bearing fields: local_id, status, models, hardware, host, port. Test extended to assert version-flipping is NOT detected and other field changes ARE detected. The no-op check is documented as best-effort, not a security boundary — a real rate limit would be separate work. 2. Replace hardcoded "ngonka" string with inferencetypes.BaseCoin. 3. Drop the redundant comment at sendTxs:574 — the function call name already says what the attempt parameter does. 4. Consolidate single-msg + batch broadcast paths. broadcastMessageAtAttempt is now a thin wrapper around broadcastMessagesAtAttempt with a 1-element slice. No behavior change; just removes the parallel near-duplicate code path that patimen flagged. 5. OOG-retry observability. When a broadcast succeeds at attempt > 0 (i.e. after the gas estimate underestimated and the doubling kicked in), log a Warn with the new gasWanted. Lets us spot in production when the static table needs re-tuning instead of relying on bug reports. 6. Pare down comments in gas_estimate.go. File header dropped from 22 lines to 8. Per-field comments kept only where load-bearing (FeeParams dependency on PoCV2 constants, authz wrap absorption in txOverheadGas). lookupMsgGas switch is now uncommented per-section since the structure is self-evident. Net diff: -33 lines despite the new no-op helper.
## Summary This PR fixes `LastUpgradeHeight` tracking for full software upgrades, exposes it through query/CLI, and validates it in the upgrade-focused Testermint flows. ## What changed ### Chain / upgrade handling - add a query for `LastUpgradeHeight` - expose it through the inference query path and autocli - stop relying on the old `BeginBlock` software-upgrade detection path - record full-upgrade height from a centralized upgrade-handler wrapper instead - add `setTrackedUpgradeHandler(...)` so new upgrade registrations use the tracked path by default - preserve that tracked registration pattern while pulling in the `v0.2.14` upgrade scaffold - update the upgrade bootstrap skill to instruct future scaffolds to use the tracked helper ### Testermint / integration coverage - add CLI plumbing to read `LastUpgradeHeight` - assert that the value is unset before upgrade boundaries - assert that partial upgrades update it to the expected heights - assert that full upgrades update it to the expected height after restart - harden local upgrade verification around known local post-upgrade API flakiness so the tests validate the feature instead of failing for unrelated reasons - make the partial-upgrade governance/test timing more realistic so the upgrade tests exercise the intended behavior ## Validation - `GOTOOLCHAIN=go1.24.2 go test ./app` - `GOTOOLCHAIN=go1.24.2 go test ./app/upgrades/v0_2_14` - targeted Testermint rerun passed: - `UpgradeTests.testVersionedEndpointSwitching()` - `UpgradeTests.submit upgrade()` The passing local rerun verified: - partial upgrades recorded `123` and then `168` - full upgrade recorded `160` - full-upgrade nodes logged `Recorded last upgrade height from upgrade handler`
…#1101) `getMustBeValidatedInferences` runs `safeUint32FromInt64` on `totalWeight`, validator weight, and executor weight before passing them to `calculations.ShouldValidate`. if the per-model summed weight crosses MaxUint32 (~4.29e9) the cast errors and the inference gets `continue`d — the claim still succeeds with an empty mustBeValidated, so the validator sidesteps validation duty without failing the message. `ShouldValidate` itself only ever does decimal math on those values, the uint32 cap is type-choice not algorithmic. widen the signature to uint64 and let the sampling proceed at the real width. `decimal.NewFromUint64` instead of `decimal.NewFromInt(int64(x))` — the latter wraps negative once x crosses MaxInt64. dapi caller in `inference_validation.go` had the same uint32 truncation on uint64 fields, fixed there too. added `safeUint64FromInt64` to keep negative-weight rejection at the call site (ValidationWeight.Weight is signed int64). regression test in keeper sets summed weight to 4.4e9 and asserts the inference IS sampled — was empty before. ground for fix is the `MaxIndividualPowerPercentage = 0.25` per-host cap from #1068. it limits per-host concentration but multiple hosts can still aggregate the per-model totalWeight past MaxUint32 as the network grows.
RewardCoins + WorkCoins gets summed as raw uint64 in three places. if the pair grows past MaxUint64 the sum wraps and you either skip a participant whose payout looks like 0, return a small wrapped total from `GetTotalCoins`, or have dapi recovery silently decide there are no unclaimed rewards. the int64 cap in `GetTotalCoins` doesn't help — wrap already happened in uint64 space. producer side `bitcoin_rewards.go` writes `^uint64(0)` into RewardCoins when the big.Int division can't fit. unreachable under default params but the sentinel wires the wrap-trap into the type. fix: cap producer to MaxInt64 instead of the MaxUint64 sentinel, switch the three sum sites to `bits.Add64` and saturate on carry. `accountsettle.go` only skips when the sum is genuinely 0, never on overflow.
`build-docker` targets hardcoded `linux/amd64`. On Apple Silicon those images don't run out of the box because Rosetta 2 doesn't translate some of the x86-64 instructions used by the BLS dependency. Each Makefile now defaults to host arch via `uname -m`. CI / cross builds pin via `DOCKER_GOARCH=amd64 make build-docker`. Affected: bridge, decentralized-api, inference-chain, proxy, proxy-ssl. Co-authored-by: John Long <john.long@productscience.ai>
## Summary - Cancels off-chain PoC validation when the validation phase ends or the PoC stage changes, and propagates cancellation into proof fetch / ML-node validation work. - Keeps retry requeue handling cancellable and avoids treating exhausted transient failures as invalid participants. This continues the work in the original `fix: add global deadline and mitigate PoC validation timeout attack` #827, with a single narrow commit and no unrelated changes. ## Why this matters Without cancellation, validation workers can keep running past the acceptance window when the PoC stage advances. The off-chain side then submits stale results (or exhausts retries on participants that simply moved on), feeding the missed-validation accounting noise that the on-chain side cannot distinguish from real misses. ## Test plan - `go test ./decentralized-api/poc/...` - Manual: trigger a stage change mid-validation, confirm the in-flight ML-node fetch is cancelled and the worker exits cleanly rather than marking the participant invalid. --- Supersedes #1124 (closed because the head fork was accidentally deleted).
… OOMing validator nodes (#1278) The BlockObserver pushes synthetic tx events into an unbounded in-memory queue with no producer backpressure and no pre-enqueue filtering, drained by only 10 workers. Under a MsgRequestThresholdSignature flood (free under MinGasPriceNgonka=0, no rate limit, no block-gas ceiling) each request amplifies into per-slot BLS work plus a blocking BroadcastTxSync on every participant; the producer outruns the 10 workers and the backing slice grows until the DAPI OOMs. Because progress is persisted only on barrier consumption, the same backlog is replayed on restart -> restart loop. Fix: - Add NewBoundedQueue(capacity) alongside NewUnboundedQueue. When full, manage() stops reading input (blocking producers) while continuing to serve output, giving clean backpressure with no busy-loop and no deadlock. NewUnboundedQueue is unchanged (capacity 0). - Construct the BlockObserver tx queue with capacity 20000. The separate NewBlock event queue stays unbounded so the height-coordination path waitForEventHeight depends on never backpressures (no deadlock cycle). - Add a producer-side relevance filter wired to the exact same predicate the consumer already uses (el.hasHandler), so it can only drop events the consumer would also discard. Barrier events bypass the filter and are always enqueued, so lastProcessedBlockHeight still advances per observed block. - Route enqueues through a ctx-aware helper that selects on ctx.Done(), so a backpressured producer shuts down cleanly. A non-enqueued (cancelled) block leaves lastQueriedBlockHeight unadvanced and is simply retried, matching existing fetch-failure semantics. Ordering, barrier semantics, and replay/progress guarantees are preserved by construction. Chain-side validation surface is unchanged.
…llateral (#1255) * fix(inference): settle accounts before releasing matured unbonding collateral Reorder onEndOfPoCValidationStage so SettleAccounts runs before the collateral module's AdvanceEpoch. Settlement evaluates participant performance for the just-completed epoch and can slash both active and unbonding collateral via SetParticipant -> UpdateParticipantStatus, while AdvanceEpoch processes the unbonding queue and releases entries whose completion_epoch equals the completed epoch. With the previous ordering, matured unbonding entries were paid out and removed before slashing could reach them, letting a participant time their unbonding to mature exactly at a slash epoch and shield that collateral from the slash. Add a regression test that pins the new ordering contract at the keeper level. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(inference): address PR review — terse comments and stable test setup Shorten inline comments in onEndOfPoCValidationStage and the collateral keeper regression test. Fix flaky interruption test setup by waiting for a stable INFERENCE node before requests run. * fix(test): use bool broker response channels in interruption setup InferenceUpAll and SetNodesActualStatus return chan bool, not a struct with Error. Fixes compile failure in API wrapper CI. --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: John Long <john.long@productscience.ai>
| for _, vw := range epochGroupData.ValidationWeights { | ||
| addr := vw.MemberAddress | ||
| ratio, ok := ratios[addr] | ||
| if !ok { | ||
| continue | ||
| } | ||
| if _, inMaint := maintenanceAddrs[addr]; inMaint { |
There was a problem hiding this comment.
Maintenance exemption here is asymmetric: it skips the INACTIVE ratio but not the ConfirmationWeight fold, so a maintenance-covered participant still loses rewards.
This block exempts maintenance-covered participants from the ConfirmationPoCRatio assignment (so they aren't marked INACTIVE), but foldEventReadings above has already lowered their ConfirmationWeight to preserved + measured, and that fold is maintenance-blind. For a participant offline during the window, measured ≈ 0 and preserved only counts whichever of their nodes happened to land in the (also maintenance-blind) preserved sample — so their ConfirmationWeight is reduced, and drops to 0 if none of their nodes are preserved for this event.
That reduction is not cosmetic. Reward effectiveWeight is directly proportional to ConfirmationWeight in bitcoin_rewards.go (effectiveWeight = ConfirmationWeight * Weight / rawTotal), so a host whose maintenance window overlaps a cPoC event still loses rewards for that epoch — up to all of them — despite the exemption. That contradicts the stated intent of this block ("expected to be offline … must not be marked INACTIVE due to maintenance-covered absence"): losing rewards is itself a penalty for the expected absence.
Suggest making the exemption symmetric — also neutralize the ConfirmationWeight lowering for maintenance-covered participants (e.g. preserve their prior ConfirmationWeight for covered epochs), not just the ratio. Maintenance is disabled by default, so this isn't a ship blocker, but it should be fixed before governance enables the feature.
There was a problem hiding this comment.
Overall, maintainance window is disabled for now. So not fthe first priority but seems like we need to fix
There was a problem hiding this comment.
Thanks for catching. It will be fixed before governance enables the feature. I'll make the exemption symmetric — preserving the prior ConfirmationWeight for maintenance-covered participants over covered epochs, not just the ratio — and add a regression test, in a separate follow-up PR.
Quick question @gmorgachev : should I target that fix at upgrade-v0.2.14, or against main / the next release branch once v0.2.14 merges? Happy to align with whatever's cleanest for the release.
There was a problem hiding this comment.
@Ryanchen911 I suggest the following fix. Can you take a look and review: a690293
There was a problem hiding this comment.
Reviewed — LGTM. This is exactly the symmetric exemption qdanik and I discussed: hoisting maintenanceAddrs into foldEventReadings as skipAddrs protects both the ConfirmationWeight fold and the ratio in one place, so a maintenance-covered host no longer loses effectiveWeight (proportional to ConfirmationWeight in bitcoin_rewards.go) for a covered epoch. Nice cleanup collapsing the now-redundant downstream if inMaint block too. The new TestFoldEventReadings_MaintenanceExemptSkipsWeightAndRatio covers the key invariant (weight preserved + no ratio for the maintenance addr, online host still folded). Thanks for picking this up. Confirming base = upgrade-v0.2.14 works for me.
There was a problem hiding this comment.
Suggestion: what if insertAt didn't mutate nodes in place, but
copied the ~depth nodes along the path instead (path copying)?
// copy the path, share everything else
newNode := &smstNode{left: node.left, right: node.right}
// ... update the one child, count, hash on newNode ...
return newNodeThen old roots stay alive, and the early-guard case becomes just:
// at the early checkpoint
s.earlyRoot = s.root // O(1), shares all unchanged subtrees with the live tree
// serving proofs against the early root
proof := buildProof(s.earlyRoot, nonce) // O(log n), no disk replayWith no rebuilds left, the snapshot cache, the rebuild-from-disk path, the
warm-up and the distinct-count limiter could all be dropped. Hashes and the
committed root format don't change, so no consensus impact, and the change to
Insert itself is tiny
And a thought for later: with a versioned tree the early guard doesn't have to
check at a fixed 1/3 - it could pick an arbitrary version (or even a few) and
verify inclusion against it, which is much harder to game than a known
checkpoint. Needs more thought...
Also worth a look for the future: https://github.com/pokt-network/smt
There was a problem hiding this comment.
Could you please experiment with such approach? Let's say 500_000 nonces, 1_000_000 nonces, 10_000_000
Looking at RAM and time to access proof first time (right now it's re-build time)
There was a problem hiding this comment.
I have another suggestion.
We currently insert nodes one by one and on each insert are recalculating affected path in a Merkle tree.
But nonces comes in a batch, and commit is sent for the batch.
When we are inserting batch there are always for each insert shared nodes in affected path (especially near root).
So it should be faster to Insert all, then rehash affected once
- Place all new leaves (structure / counts) in a batch, mark nodes on their paths dirty
- Rehash each dirty node once, bottom-up
I think this optimization could be easily added
There was a problem hiding this comment.
And a thought for later: with a versioned tree the early guard doesn't have to
check at a fixed 1/3 - it could pick an arbitrary version (or even a few) and
verify inclusion against it, which is much harder to game than a known
checkpoint. Needs more thought...
Don't we already have this ability? We can ask for proof on any count and tree will be build for the requested count by stored ordered artifacts
There was a problem hiding this comment.
I have another suggestion.
We currently insert nodes one by one and on each insert are recalculating affected path in a Merkle tree. But nonces comes in a batch, and commit is sent for the batch.
When we are inserting batch there are always for each insert shared nodes in affected path (especially near root).
So it should be faster to Insert all, then rehash affected once
- Place all new leaves (structure / counts) in a batch, mark nodes on their paths dirty
- Rehash each dirty node once, bottom-up
I think this optimization could be easily added
Yes, agree, the same mechanic (along with all the suggestions above) is already implemented #1432
There was a problem hiding this comment.
And a thought for later: with a versioned tree the early guard doesn't have to
check at a fixed 1/3 - it could pick an arbitrary version (or even a few) and
verify inclusion against it, which is much harder to game than a known
checkpoint. Needs more thought...Don't we already have this ability? We can ask for proof on any count and tree will be build for the requested count by stored ordered artifacts
Technically any count is possible, but in practice it is capped because cache misses cause O(N) rebuilds and the limiter protects against dos (in the current implementation)
There was a problem hiding this comment.
@gmorgachev Are we ready to merge #1432 into this upgrade?
There was a problem hiding this comment.
from my mind, merging it into this upgrade could be risky, so it needs additional testing. I'd rather ship it in the next update
| } | ||
|
|
||
| func (v *OffChainValidator) maybeWarmEarlySnapshot(epochState chainphase.EpochState) { | ||
| stage, target, ok := EarlyShareCaptureTarget(&epochState, earlyshare.DefaultFirstFraction) |
There was a problem hiding this comment.
I think it should be v.guard.FirstFraction() instead of earlyshare.DefaultFirstFraction?
| func ApplyMissStreak(prev GuardState, passed bool, isConfirmation bool, stageHeight int64) MissOutcome { | ||
| next := prev | ||
| next.UpdatedStageHeight = stageHeight | ||
|
|
||
| if passed { | ||
| if isConfirmation { | ||
| // Only a confirmation PoC pass is trusted to clear the streak. | ||
| next.ConsecutiveMisses = 0 | ||
| } | ||
| // Regular PoC pass: leave the streak untouched. | ||
| return MissOutcome{VoteNo: false, NewState: next} | ||
| } | ||
|
|
||
| // Failure in either phase always accrues a miss. | ||
| next.ConsecutiveMisses = prev.ConsecutiveMisses + 1 |
There was a problem hiding this comment.
I would add an early return when prev.UpdatedStageHeight == stageHeight to keep miss-streak updates idempotent in all scenarios as a proactive safety fix
| prev, _, err := g.store.GetGuardState(ctx, addr, modelID) | ||
| if err != nil { | ||
| logging.Warn("EarlyShareGuard: guard-state load failed; skipping participant", types.PoC, | ||
| "stage", stageHeight, "participant", addr, "model", modelID, "error", err) | ||
| continue | ||
| } | ||
| outcome := earlyshare.ApplyMissStreak(prev, passed, isConfirmation, stageHeight) | ||
| if err := g.store.UpsertGuardState(ctx, outcome.NewState); err != nil { |
There was a problem hiding this comment.
Would it make sense to reset (or ignore) miss streaks accumulated in observe when switching to enforce?
There was a problem hiding this comment.
I think it's ok to not reset. Reset mechanic probably requires a new postrgres relation, to persist the prev value. In my opinion it's too much state handling for this case
the default setting in enforced anyway + I don't think it's obvious that we need to reset the counts from the observe mode, it's still valuable history
There was a problem hiding this comment.
Made sure the default setting is enforce: 5621f45
| if len(checkpoints) > 0 { | ||
| if err := g.store.UpsertCheckpoints(ctx, checkpoints); err != nil { | ||
| logging.Error("EarlyShareGuard: failed to persist early checkpoints", types.PoC, "stage", stageHeight, "error", err) | ||
| return | ||
| } | ||
| } | ||
| if err := g.store.MarkStageCaptured(ctx, stageHeight, target, capturedAt, perModel); err != nil { | ||
| logging.Error("EarlyShareGuard: failed to mark capture run", types.PoC, "stage", stageHeight, "error", err) |
There was a problem hiding this comment.
Should an empty capture be treated as failed instead of completed, so it can retry on the next block?
There was a problem hiding this comment.
the capture query is pinned to the target block height. The chain's state at a given height is frozen forever. So retrying won't yield any different results
| // DefaultStaleTTL is the node-retirement age: an unobserved node is dropped | ||
| // after this long, and only while its model still has a live flow. Must be | ||
| // >= the fresh window. | ||
| const DefaultStaleTTL = time.Hour |
There was a problem hiding this comment.
The devshardd ML-node cache (staleTTL 1h + flow-gated pruning) keeps serving indefinitely while DAPI is down, but the fallback path never counts in-flight requests, so the broker's MaxConcurrent ceiling is bypassed.
When devshardd lives without dapi it can hammer its ML nodes with unbounded concurrency.
The cache was meant to survive DAPI upgrades/short reboots — a long-lived cache is over-engineered for that goal and just widens this window
It's informational note, and it is not proposed to change anything, as I think, this situation will never happen when both intersects: dapi is down and a lot of concurrent inferences/validations
There was a problem hiding this comment.
but the fallback path never counts in-flight requests
could you add this one?
i made it work indefinitely for the long term strategy to be close to 99.9% SLA
There was a problem hiding this comment.
I think it could be added to next release, and is not a blocker
I will add it
There was a problem hiding this comment.
@gmorgachev
I've finished the review and if we do not start voting on this release in a few hours, I can add this today. Anyway I will prepare PR soon
There was a problem hiding this comment.
Done
#1443
I think it is safe to merge without testing, as it only adds functionality for decentralized-api and doesn't touch life code paths. Devshard part can go in separate devshard-only release.
|
Three pre-existing gaps (not from this PR), none looks like a real attack on its
None of these looks like a real attack vector on its own but they're not written down anywhere, |
it should vote invalid if can't get
i think it's correct, as we are checking against 2/3 of weight, not against 2/3 - validatorWeight |
…tcome, not host fault (#1434) * fix(devshard): classify Kimi reasoning-burn empty streams as model outcome, not host fault * normalize model id in logs Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Signed-off-by: DimaOrekhovPS <dima.orekhov@productscience.ai> --------- Signed-off-by: DimaOrekhovPS <dima.orekhov@productscience.ai> Co-authored-by: DimaOrekhovPS <dima.orekhov@productscience.ai> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
It was discussed multiple times at H1. |
|
There was a host leak fix (400GB RSS production incident) Commit: ae2b525 — merge cache manager (#1417) For every escrow it is creating 20 go routings for validation, if there are N escrows there are 20*N validation go routines. But anyway concurrent execution is limited by maxConcurrent at ML node manager level, and mostly all routines are at stale state as they are awaiting results from ML nodes. I think it's better to use shared validation routines pool. Share across all escrows, for one devshardd process. It's also not a blocker and could be added to next release |
|
Routes and chain msgs are disabled for old inference path, but related files and chain queries are kept. Maybe we should cleanup them, or this is deferred and scheduled to next releases? |
|
decentralized-api isn't listening I think it's easy to add it to current release, so gateway can poll it from dapi to support maintenance at gateway release and don't wait for next 0.2.15 release |
Yes, this upgrade is the first step |
|
The chain already exposes this via query endpoints, not just events — You're right that decentralized-api itself doesn't subscribe to the @a-kuprin which is it on the gateway side — can it hit the chain query API directly, or does it need a dapi-side endpoint? |
My suggestion in the comment above was that maybe we should document these three assumptions explicitly, since they’re easy to miss during review - not to question why it works this way or to change it |
Yes I've seen the exposed query endpoints and it works. Target design is proposed here: #1367 It's not an issue, but some polishing |
) * perf(devshard): shrink gateway memory footprint and add profiling Reduce devshardctl gateway RSS by not holding inactive escrows in memory, bounding the chat response cache, and avoiding full inference-map deep copies on the common read paths. Adds pprof/memstats for diagnosis. Runtime lifecycle: - Boot loads only devshards marked active; inactive ones are skipped instead of being hydrated (and re-queried on chain) at startup. - Non-resident devshards are served on demand: read-only debug/status routes hydrate a transient local-SQLite-only runtime (no chain/host clients), and settlement rehydrates a transient full runtime; both are released right after use. - All disable paths (admin deactivate, rotation-without-settle, settle) now retire the runtime from memory drain-safely, only after in-flight requests complete. State accessors (avoid copying the full inference map): - Add StateMachine.Config, SnapshotStateNoInferences, SnapshotInferences, and InferenceStatusCounts; use them at status/config/single-inference call sites instead of SnapshotState. Endpoints: - /v1/state is now summary-only; the full inference dump moves to a new admin/debug endpoint /v1/debug/inferences (no pagination). - /v1/debug/state reports inference status counts; single-inference lookup uses GetInference. - Add /v1/debug/memstats and register net/http/pprof under /debug/pprof/, both gated behind admin auth. Chat response cache: - Fix unbounded growth: entries keyed by request-body hash were only expired lazily on lookup, so unique requests lived until restart. Add a periodic expiry sweep and a total-bytes cap (DEVSHARD_CHAT_CACHE_MAX_BYTES). * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Signed-off-by: DimaOrekhovPS <dima.orekhov@productscience.ai> * fix(devshard): route /v1/debug/inferences in single-runtime mode and update testermint PR #1435 moved the full inference dump from /v1/state to the new /v1/debug/inferences endpoint, but only registered it on the per-devshard runtime mux. Bare-path (single-escrow) callers got a 404, and testermint's getDevshardProxyInferences still read the now-absent "inferences" field from /v1/state, so DevshardStandaloneTests saw zero finished inferences. - Register /v1/debug/inferences on the gateway mux via handleSingleOnly, matching the other per-runtime debug routes. - Point testermint's getDevshardProxyInferences at /v1/debug/inferences. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Signed-off-by: DimaOrekhovPS <dima.orekhov@productscience.ai> Co-authored-by: David and Daniil Liberman <da@liberman.net> Co-authored-by: DimaOrekhovPS <dima.orekhov@productscience.ai> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com>
* Fix PoC retry exhaustion vote Co-authored-by: Cursor <cursoragent@cursor.com> * don't slash participants on maintenance * Don't cancel PoC validation on transient not-synced tracker state A momentary not-synced phase tracker reading (RPC lag, catch-up) made shouldStopValidationForStage return true, permanently cancelling all in-flight validation for the stage. Treat not-synced like nil: wait for the next tick and only cancel on a synced reading showing a real phase end or stage change. Co-authored-by: Cursor <cursoragent@cursor.com> * Warm early snapshot at the configured first fraction maybeWarmEarlySnapshot used the hardcoded DefaultFirstFraction while the guard captures at the configured FirstFraction, so a non-default config warmed the snapshot cache at the wrong checkpoint height. Use the guard's nil-safe FirstFraction() for both. Co-authored-by: Cursor <cursoragent@cursor.com> * Ship the early-share guard in enforce mode by default Nodes that upgrade the binary without config changes must enforce the guard. The config loader pre-seeds DefaultConfig before yaml/env, so flipping the default mode to enforce activates it everywhere except nodes with an explicit mode override. Co-authored-by: Cursor <cursoragent@cursor.com> * Make early-share miss streaks idempotent --------- Co-authored-by: Cursor <cursoragent@cursor.com>
|
I have created follow-up PR related to PR comments: I think it is safe to merge without testing, as it only adds functionality for decentralized-api and doesn't touch life code paths. Devshard part can go in separate devshard-only release. Added:
Before devshardd is initing lazily on first served inference, and it asks chain node for escrow data. When escrow is created but no inferences was sent, and node is down (or during upgrade) if first inference to devshard will be send it will fail. Also added discussed:
|
* devshard: stamp rotation-created escrows with the gateway protocol version Escrows minted by the auto-rotator (and depletion replacement) were persisted without a protocol version, which defaults to v1 — so any gateway with rotation enabled drifted back to v1 escrows regardless of the protocol its escrows were registered with. Derive the protocol for rotation-created escrows from the gateway-wide route prefix (DEVSHARD_ROUTE_PREFIX / build version): a gateway serving /devshard/v3 now mints protocol-v3 escrows. The protocol is frozen into the write-ahead commitment record (with a sqlite column migration) so escrows recovered via reconcile keep it. Unparseable route versions fall back to the previous v1-default behavior. Co-authored-by: Cursor <cursoragent@cursor.com> * devshard: review follow-ups for rotation protocol version Warn-log each fallback path in rotationEscrowProtocolVersion instead of silently defaulting, and map semver-like route versions by their major component (v2.1.0 -> v2) so semver-named runtimes still get a protocol stamp. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>
…1456) TestNewRotationDevshardStateDoesNotForceProtocolVersion was added when the hardcoded ProtocolV2 stamp from the gateway-v2 branches was removed, to guard against that constant sneaking back in via cherry-picks. Since rotation now derives the protocol from the gateway route prefix, the empty-field assertion only held because tests run without DEVSHARD_ROUTE_PREFIX. Rename the test to state the current contract and assert both sides: empty without a resolvable route version, derived (not hardcoded) with one. Co-authored-by: Cursor <cursoragent@cursor.com>
#1432] (#1447) * perf(poc/artifacts): defer SMST hashing to flush — byte-identical roots, up to 3.4x faster ingest insertAt re-hashed every node on the root->leaf path on each insert, so a shared upper node was re-hashed once per descendant insert. Record structure and counts on insert (node.hash = nil) and fill hashes lazily once when the root is needed (GetRoot / snapshot proof / rebuild) via ensureHashed, hashing each node once per flush instead of once per insert that passes through it. Roots and proofs are byte-identical to per-insert hashing (same Merkle function, different hashing order): a 200k-insert fingerprint over flush roots and sampled proofs matches the previous implementation exactly, and per-N roots match at 100k / 500k / 1M / 10M. Hashing mutates nodes, so live-tree read paths that may fill hashes take the write lock; the live-tip proof path keeps a read-lock fast path once the hashes are already filled. Adds TestDeferredHashFingerprint (root/proof determinism guard), live-tip and historical-rebuild proof-path tests, and an env-gated build profile (TestSMSTBuildProfile) reproducing the ingest/RAM numbers. * perf(poc/artifacts): copy-on-write SMST — O(depth) historical proofs, drop the rebuild-DoS limiter Historical snapshot proofs rebuilt the whole tree from the artifact log on every request (O(N) SHA-256), which a validator could turn into a rebuild-DoS by cycling distinct counts — guarded until now by a per-validator distinct-count quota. This makes the live tree copy-on-write and serves committed-count proofs from retained snapshots in O(depth). - insertCOW rewrites only the root->leaf path and shares untouched subtrees, so a captured root stays valid. Roots and proofs are byte-identical to Insert (same Merkle function, hashing deferred identically); the on-chain commitment is unchanged. - The store captures a retained snapshot at every committed (flush) count and serves its proofs from shared nodes in O(depth). On restart, recover() re-captures a snapshot at each committed count in one copy-on-write replay, so historical proofs stay O(depth) after a restart. - A proof at a non-committed count is rejected outright (its root can never match the on-chain commitment), so a proof request can no longer trigger a rebuild. The rebuild path remains only as a defensive cold-start fallback. - With the rebuild-DoS surface gone, the per-validator distinct-snapshot-count limiter is removed. Byte-identical roots/proofs verified across the suite; live, retained-historical and post-restart proof paths plus the concurrent-readers test are green under -race. * test(poc/artifacts): differential Insert==COW==rebuild + finding regressions Byte-identity of eager Insert, copy-on-write insert, and rebuild-from-log across roots and raw proofs at many N (through depth-expansion boundaries); served transport proofs verify against the eager root. Plus two regression tests for the load-test findings this PR addresses: a non-committed proof count is rejected outright (no O(N) rebuild), and an early committed root is served from a re-captured retained snapshot in O(depth) after a restart, byte-identical to the pre-restart root. * fix(poc/artifacts): release write lock before retained proof I/O * fix(poc/artifacts): persist flush roots independent of distributions.jsonl Recover used distributionHistory counts as the only durable flush markers, so a warn-only dist append failure made an on-chain flush look non-committed after restart. Write count+root to flushed_roots.jsonl on flush, union it on recover, backfill for pre-roots stores, and test surviving a wiped distributions file. * fix(poc/artifacts): fill live-tip hashes under Lock, serve proofs under RLock * feat(poc/artifacts): env toggles for deferred hashing and COW + 30-flush profiles SMST_DEFERRED_HASH and SMST_COW default on; set either to 0 for profiling baselines. When COW is off, pin flushed counts via the process snapshot cache (WarmSnapshot/PrebuildSnapshot) so early 1/3 commits stay cached without retained trees. Add a 30-flush early-snapshot profile and coverage for defaults and COW-off early cache behavior. * feat(poc/artifacts): tip Prebuild in-memory clone vs artifact rebuild toggle When COW is off, PrebuildSnapshot can deep-clone under the write lock (default) or rebuild from artifacts without holding the lock (SMST_SNAPSHOT_IN_MEMORY_CLONE=0, v0.2.14-style). Refresh follow-up PR notes and Flush30 profile matrix. * feat(poc/artifacts): multicore ensureHashed behind SMST_PARALLEL_HASH Fan out deferred hash fill across GOMAXPROCS (default on); eager path hashing stays serial. Add root-equality coverage and deferred×parallel profile matrix; refresh follow-up PR notes. --------- Co-authored-by: kAIPraxisBot <272124397+kAIPraxisBot@users.noreply.github.com> Co-authored-by: akup <ak@neonavigation.com> Co-authored-by: DimaOrekhovPS <dima.orekhov@productscience.ai>
#1457) * Keep previous epoch validators when PoC validation yields an empty set An epoch transition where nobody passed PoC validation previously produced an empty epoch: no active participants, no epoch group members, no voting powers for the next round, permanently stalling the network. Fall back to re-seating the current epoch's still-valid validators (excluding anyone removed, excluded, or invalidated mid-epoch) as freshly initialized participants, and refuse to hand the staking module a validator set with no positive power. Co-authored-by: Cursor <cursoragent@cursor.com> * replace epoch index in reused RandomSeed Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Signed-off-by: DimaOrekhovPS <dima.orekhov@productscience.ai> * Add Testermint coverage for empty-PoC epoch validator fallback Verify that when PoC yields no active participants the seatbelt re-seats the previous live validators instead of forming an empty epoch. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Signed-off-by: DimaOrekhovPS <dima.orekhov@productscience.ai> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
15 participants
Upgrade Proposal: v0.2.14
This PR prepares the v0.2.14 release.
The mainnet chain/API work focuses on PoC duplicate-artifact protection, early share detection, classic inference API deprecation (disabling
/v1/chat/completionsbilling on mainnet and removing embedded/v1/devshardfrom the API binary), reward recipient routing, and upgrade-time safety fixes.The devshard part prepares the v3 runtime so brokers can serve inference during the chain upgrade without depending on the deprecated classic API path, improving RAM utilization and enabling safe switching between SQLite and Postgres storage.
Upgrade Plan
The node binary is upgraded through an on-chain software upgrade proposal. Existing hosts are not required to manually update their
apiornodecontainers as part of the upgrade.A separate devshard v3 release from this branch will be proposed and rolled out before the mainnet chain upgrade. Brokers who switch inference traffic to
/devshard/v3ahead of time can keep serving inference while the chain upgrade runs.Proposed Process
/devshard/v3.Changes
inference-chain / decentralized-api
9a63468,22a21b0by @gmorgachev, @DimaOrekhovPS, @0xMayoor.devshard
devsharddto continue serving during DAPI outages using an ML-node cache, add per-escrow SQLite/Postgres storage routing, and snapshotvalidation_rateat escrow creation #1417 by @akup.Testing
The devshard v3 runtime was deployed and verified first. During testing,
nodeandapicontainers were stopped while devshard traffic was running; active requests were allowed to finish, and new requests were successfully created and executed through/devshard/v3.After devshard v3 validation, the mainnet-style upgrade from
v0.2.13tov0.2.14was tested.Contributors (sorted alphabetically)
Proposed Bounties