# perf(devshard): port escrow runtime lifecycle + gateway memory optimizations from dl/gateway-v2-mem#1402
Open
libermans wants to merge 5 commits into
Conversation
…#1314) * feat(devshard): settle depleted escrow when last active request drains * fix(devshard): skip settlement reconcile on restart when settlement disabled
* fix(devshard): retire runtime escrow * review(devshard): copilot suggestions & todo for future task implementation * fix(devshard): retire runtime escrow when in-fligh request completed
Reduce devshardctl gateway RSS by not holding inactive escrows in memory, bounding the chat response cache, and avoiding full inference-map deep copies on the common read paths. Adds pprof/memstats for diagnosis. Runtime lifecycle: - Boot loads only devshards marked active; inactive ones are skipped instead of being hydrated (and re-queried on chain) at startup. - Non-resident devshards are served on demand: read-only debug/status routes hydrate a transient local-SQLite-only runtime (no chain/host clients), and settlement rehydrates a transient full runtime; both are released right after use. - All disable paths (admin deactivate, rotation-without-settle, settle) now retire the runtime from memory drain-safely, only after in-flight requests complete. State accessors (avoid copying the full inference map): - Add StateMachine.Config, SnapshotStateNoInferences, SnapshotInferences, and InferenceStatusCounts; use them at status/config/single-inference call sites instead of SnapshotState. Endpoints: - /v1/state is now summary-only; the full inference dump moves to a new admin/debug endpoint /v1/debug/inferences (no pagination). - /v1/debug/state reports inference status counts; single-inference lookup uses GetInference. - Add /v1/debug/memstats and register net/http/pprof under /debug/pprof/, both gated behind admin auth. Chat response cache: - Fix unbounded growth: entries keyed by request-body hash were only expired lazily on lookup, so unique requests lived until restart. Add a periodic expiry sweep and a total-bytes cap (DEVSHARD_CHAT_CACHE_MAX_BYTES).
This was referenced Jul 6, 2026
Collaborator
|
PR mentions #1399 |
a-kuprin
reviewed
Jul 7, 2026
a-kuprin
reviewed
Jul 7, 2026
Centralize settlement-pending cleanup in settleDevshardOnChain so admin settles clear the flag and avoid re-broadcast on restart. Remove unused SnapshotInferences and propagate SQLite upsert lookup errors except sql.ErrNoRows.
Collaborator
Now it is included |
a-kuprin
approved these changes
Jul 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Cherry-picks three commits from
dl/gateway-v2-mem(#1399) to reduce devshardctl gateway RSS, adapted where this branch's architecture diverges:feat(devshard): settle depleted escrow after in-flight requests drain(feat(devshard): settle depleted escrow after in-flight requests drain #1314, clean pick) — depleted escrows are marked settlement-pending and settled once the last active request drains; pending settlements are reconciled on restart.fix(devshard): retire runtime escrow(fix(devshard): retire runtime escrow #1349, clean pick) — deactivation paths drop the runtime from memory drain-safely, releasing its state machine, inference map, and SQLite handles.perf(devshard): shrink gateway memory footprint and add profiling(from perf(devshard): shrink gateway memory footprint and add profiling #1399, conflicts resolved) — boot loads only active devshards; non-resident escrows are served via transient rehydration (read-only for debug/status, full for settlement);/v1/statebecomes a bounded summary with the inference dump moved to/v1/debug/inferences; adds/v1/debug/memstats+ admin-gated pprof; fixes the unbounded chat response cache (periodic expiry sweep + total-bytes cap,DEVSHARD_CHAT_CACHE_MAX_BYTES).Adaptations for this branch
The memory commit predates this branch's sealed-inference architecture (live records are evicted to SQLite on seal), so it was not applied verbatim:
SnapshotStateNoInferencesreferencedRevealedSeeds, a field that doesn't exist in this branch'sEscrowState— auto-merged cleanly but wouldn't compile; removed. Its shallow copy sharesSealedAcc, which is safe (only ever replaced wholesale, never mutated)./v1/debug/inferencesuses sealed-awareExportAllInferenceRecordsinstead of the PR's live-onlySnapshotInferences, so sealed records still appear in the dump./v1/debug/statekeeps this branch's phase field and live/sealed count split while adopting the new no-deep-copy accessors (InferenceStatusCounts,LatestNonce,Balance).handleInference(X-Inference-Id) handler — this branch removed that endpoint and nothing routes to it.session_close_test.goandflush_snapshot_test.goupdated to this branch'sstorage.Storageinterface and test helpers (newTestStateMachine,RuntimeTestVersion);TestHandleState_IncludesSealedInferencesnow asserts sealed records via/v1/debug/inferencesand that/v1/statestays bounded.Breaking change
/v1/stateno longer includes theinferencesmap — consumers must read/v1/debug/inferences(admin/debug, unpaginated).Not included
b87e51b(suspicious-host tracking) — this branch already carries a diverged version of that work.escrow_rotator.go, disjoint functions).handleAdminSettleDevsharddoesn't clearSettlementPendingon success, so an admin-settled escrow is re-broadcast on every restart. Worth a follow-up one-liner.Testing
go test ./...passes (all packages, including the picks' new tests:gateway_inactive_access_test.go,flush_snapshot_test.go,session_close_test.go).go vet ./...clean.