fix(runtime): prevent cumulative heartbeat memory growth - #7244
Conversation
|
Hi @rayrayraykk, this is your 259th Pull Request. 📋 About PR TemplateTo help maintainers review your PR faster, please make sure to include:
Complete PR information helps speed up the review process. You can edit the PR description to add these details. 🙌 Join Developer CommunityThanks so much for your contribution! We'd love to invite you to join the official QwenPaw developer group! You can find the Discord and DingTalk group links under the "Developer Community" section on our docs page: We truly appreciate your enthusiasm—and look forward to your future contributions! 😊 We'll review your PR soon. |
There was a problem hiding this comment.
Pull request overview
This PR fixes unbounded backend memory growth during idle/stalled streaming by changing runtime heartbeat emissions to a lightweight keepalive event instead of re-serializing the full, cumulative AgentResponse on every tick. This prevents TaskTracker’s per-run SSE replay buffer from retaining gigabytes of duplicated response snapshots during long “Thinking” stalls.
Changes:
- Updated
Envelope.heartbeat()to emit a minimalEvent(object="message", type="heartbeat")rather than the mutable accumulated_response. - Added regression tests to ensure heartbeats stay small and do not mutate or repeat accumulated turn output.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
src/qwenpaw/runtime/envelope.py |
Replaces cumulative full-response heartbeat snapshots with a lightweight heartbeat event to prevent SSE buffer memory blowups. |
tests/unit/runtime/test_envelope_heartbeat.py |
Adds unit tests verifying heartbeat payload size and that accumulated response state is unchanged. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Description
Fix unbounded backend memory growth while a long-running Console turn is
idle after accumulating large tool results, especially screenshot-heavy
desktop automation turns.
Root cause
This is a multiplier that requires both a large current-turn response and an
idle/stalled upstream stream:
Envelope._response.outputaccumulates completed messages and toolresults from the current turn. Frozen screenshots can be close to 2 MiB
before Base64 encoding.
AgentExecutoremits a heartbeat every 25 seconds while AgentScope is notyielding an event (for example, while a SiliconFlow DeepSeek stream is
stalled in Thinking).
Envelope.heartbeat()previously yielded the entire mutableAgentResponse.every heartbeat.
TaskTracker._RunState.bufferretains every SSE string until the run endsso reconnecting clients can replay it.
The retained size is therefore approximately:
serialized current-turn response size × heartbeat countAs the turn itself also grows, the total can become super-linear over the
whole run. This matches reports where the backend remains in Thinking,
repeated event serialization warnings appear, and the process eventually
raises
MemoryError.QwenPaw 2.1.0 had no semantic stalled-stream watchdog, so the multiplier
could run indefinitely. 2.1.1b1 contains the stream watchdog from #7150,
but an OpenAI-compatible provider can still wait before returning the stream
object (the OpenAI SDK default read timeout is 600 seconds and retries may
apply). The upstream stall is the trigger; the cumulative heartbeat is the
unbounded memory multiplier fixed here.
Fix
Emit the frontend's existing lightweight heartbeat event:
{"object":"message","type":"heartbeat"}instead of re-emitting
_response.No model delta, thinking block, ToolChunk/tool result, completed message, or
final
AgentResponseis dropped. Only redundant full-response snapshots usedas transport keepalives are replaced. The Console already treats this event
as a no-op, and reconnect replay still retains all real stream events.
Reproduction
Two complementary reproductions were used.
1. Local SiliconFlow/DeepSeek-compatible stalled API
A local
/v1/chat/completionsendpoint emitted OpenAI-compatible DeepSeekreasoning chunks and then deliberately kept the SSE connection open:
The harness used the real OpenAI SDK, AgentScope
Agent, QwenPawOpenAIChatModelCompat,RetryChatModel,AgentExecutor,Envelope, SSEserialization, and
TaskTracker. Eight 2 MiB screenshot-shaped tool resultswere accumulated in the same Envelope before the stalled model step.
For test speed, the heartbeat interval was reduced from 25 seconds to 0.25
seconds. The object serialized and retained on each tick was unchanged.
Twenty-four ticks correspond to ten minutes at the production interval.
This also shows that stop works when the process is healthy. At multi-GB
memory pressure, repeated large JSON serialization and allocation can make
both the UI and backend appear unresponsive before cancellation is handled.
2. Windows process-memory benchmark
The Windows benchmark uses the real QwenPaw schema and Envelope, six frozen
screenshot-shaped outputs at the 2 MiB raw-image limit, and the same SSE
strings retained by
TaskTracker. It serializes 160 heartbeat events, equalto 66 minutes 40 seconds at the production interval.
Reproduction script and successful
windows-latestrun:Environment: Windows Server 2025, CPython 3.12.10.
The issue is cross-platform Python retention, not a WebView leak. Windows
Task Manager reports
qwenpaw-backend.exeand WebView2 as separate rows;Windows was used here because it matches the reported environment and exposes
both Working Set and Private Bytes.
Evidence
The commit contains only the runtime fix and its regression tests. Benchmark
and mock-service files are intentionally not part of the PR.
The regression test verifies that a heartbeat does not contain a 1 MiB
accumulated output, stays below 256 bytes, and does not mutate the accumulated
response that is emitted normally at finalization.