perf(devshard): shrink gateway memory footprint and add profiling#1399
Open
libermans wants to merge 2 commits into
Open
perf(devshard): shrink gateway memory footprint and add profiling#1399libermans wants to merge 2 commits into
libermans wants to merge 2 commits into
Conversation
Reduce devshardctl gateway RSS by not holding inactive escrows in memory, bounding the chat response cache, and avoiding full inference-map deep copies on the common read paths. Adds pprof/memstats for diagnosis. Runtime lifecycle: - Boot loads only devshards marked active; inactive ones are skipped instead of being hydrated (and re-queried on chain) at startup. - Non-resident devshards are served on demand: read-only debug/status routes hydrate a transient local-SQLite-only runtime (no chain/host clients), and settlement rehydrates a transient full runtime; both are released right after use. - All disable paths (admin deactivate, rotation-without-settle, settle) now retire the runtime from memory drain-safely, only after in-flight requests complete. State accessors (avoid copying the full inference map): - Add StateMachine.Config, SnapshotStateNoInferences, SnapshotInferences, and InferenceStatusCounts; use them at status/config/single-inference call sites instead of SnapshotState. Endpoints: - /v1/state is now summary-only; the full inference dump moves to a new admin/debug endpoint /v1/debug/inferences (no pagination). - /v1/debug/state reports inference status counts; single-inference lookup uses GetInference. - Add /v1/debug/memstats and register net/http/pprof under /debug/pprof/, both gated behind admin auth. Chat response cache: - Fix unbounded growth: entries keyed by request-body hash were only expired lazily on lookup, so unique requests lived until restart. Add a periodic expiry sweep and a total-bytes cap (DEVSHARD_CHAT_CACHE_MAX_BYTES).
Contributor
Author
|
@qdanik somehow I can't add you to the reviewer list |
Contributor
|
@libermans there is only one minor issue, not a blocker.
|
qdanik
approved these changes
Jul 4, 2026
a-kuprin
reviewed
Jul 7, 2026
a-kuprin
approved these changes
Jul 7, 2026
Centralize settlement-pending cleanup so admin settles do not leave a stale flag that retriggers on restart. Update settlement test mocks to mirror the real post-broadcast bookkeeping. Co-authored-by: Cursor <cursoragent@cursor.com>
Collaborator
Fixed |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reduce devshardctl gateway RSS by not holding inactive escrows in memory, bounding the chat response cache, and avoiding full inference-map deep copies on the common read paths. Adds pprof/memstats for diagnosis. Runtime lifecycle: