fix(devshard): bound the gateway chat response cache#1409
Closed
ouicate wants to merge 1 commit into
Closed
Conversation
The gateway's process-wide chatResponseCache dedups chat completions keyed by a hash of model + request body, with a 1h TTL. Expired entries are only reclaimed lazily in Get -- i.e. when the same key is looked up again. Real chat traffic has near-unique bodies, so a key is essentially never queried twice: the expired entry is never revisited and never deleted, and there was no size cap. The map grew by one full response body per distinct request and never shrank -- an unbounded memory leak that scales with traffic x uptime. Cap the cache at defaultChatResponseCacheMaxEntries (2048) and evict when a new key would exceed it. Eviction runs under the existing lock in two phases: reclaim all expired entries first (usually enough for a unique-body workload), then, only if still at capacity, drop the oldest half in one pass. Batching the drop amortizes the O(n) sweep and always keeps the newest entries so a just-cached response survives an immediate retry. This matches the bounded-map idiom already used by the rate limiter and gossip dedup.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The gateway keeps a single process-wide
chatResponseCache(devshard/cmd/devshardctl/response_cache.go) that dedups chat completions, keyed by a SHA-256 ofmodel + request bodywith a 1h TTL. Expired entries are only ever reclaimed lazily insideGet— i.e. when the same key is looked up again.Real chat traffic has near-unique request bodies (different messages/ids), so a given key is essentially never queried a second time. Its expired entry is therefore never revisited and never deleted, and there is no size cap. The map grows by one full cached response body per distinct request and never shrinks: an unbounded memory leak that scales with total traffic × uptime, independent of how many entries are actually live.
Fix
Cap the cache at
defaultChatResponseCacheMaxEntries(2048) and evict when a new key would exceed it. Eviction runs under the existing lock in two phases:ExpiresAt, which is insertion time + constant TTL, i.e. insertion order).Dropping half amortizes the O(n) sweep across ~maxEntries/2 inserts instead of paying it on every
Setonce hot, and always keeps the newest entries so a just-cached response survives an immediate retry. Overwriting an existing keyskips eviction (the count doesn't grow). This matches the bounded-map idiom already used by the rate limiter and gossip dedup.
Tests
response_cache_test.go: set/get round-trip with body-copy isolation, expired entry dropped on access, hard bound under 500 unique inserts (never exceeds max, newest survives), expired-reclaimed-before-live eviction priority, andoverwrite-does-not-evict.