Skip to content

fix(runtime): prevent cumulative heartbeat memory growth - #7244

Merged
rayrayraykk merged 1 commit into
agentscope-ai:mainfrom
rayrayraykk:investigate/windows-backend-memory-growth
Aug 24, 2026
Merged

fix(runtime): prevent cumulative heartbeat memory growth#7244
rayrayraykk merged 1 commit into
agentscope-ai:mainfrom
rayrayraykk:investigate/windows-backend-memory-growth

Conversation

@rayrayraykk

Copy link
Copy Markdown
Member

Description

Fix unbounded backend memory growth while a long-running Console turn is
idle after accumulating large tool results, especially screenshot-heavy
desktop automation turns.

Root cause

This is a multiplier that requires both a large current-turn response and an
idle/stalled upstream stream:

  1. Envelope._response.output accumulates completed messages and tool
    results from the current turn. Frozen screenshots can be close to 2 MiB
    before Base64 encoding.
  2. AgentExecutor emits a heartbeat every 25 seconds while AgentScope is not
    yielding an event (for example, while a SiliconFlow DeepSeek stream is
    stalled in Thinking).
  3. Envelope.heartbeat() previously yielded the entire mutable
    AgentResponse.
  4. The Console serialized that cumulative response into a new SSE string on
    every heartbeat.
  5. TaskTracker._RunState.buffer retains every SSE string until the run ends
    so reconnecting clients can replay it.

The retained size is therefore approximately:

serialized current-turn response size × heartbeat count

As the turn itself also grows, the total can become super-linear over the
whole run. This matches reports where the backend remains in Thinking,
repeated event serialization warnings appear, and the process eventually
raises MemoryError.

QwenPaw 2.1.0 had no semantic stalled-stream watchdog, so the multiplier
could run indefinitely. 2.1.1b1 contains the stream watchdog from #7150,
but an OpenAI-compatible provider can still wait before returning the stream
object (the OpenAI SDK default read timeout is 600 seconds and retries may
apply). The upstream stall is the trigger; the cumulative heartbeat is the
unbounded memory multiplier fixed here.

Fix

Emit the frontend's existing lightweight heartbeat event:

{"object":"message","type":"heartbeat"}

instead of re-emitting _response.

No model delta, thinking block, ToolChunk/tool result, completed message, or
final AgentResponse is dropped. Only redundant full-response snapshots used
as transport keepalives are replaced. The Console already treats this event
as a no-op, and reconnect replay still retains all real stream events.

Reproduction

Two complementary reproductions were used.

1. Local SiliconFlow/DeepSeek-compatible stalled API

A local /v1/chat/completions endpoint emitted OpenAI-compatible DeepSeek
reasoning chunks and then deliberately kept the SSE connection open:

@app.post("/v1/chat/completions")
async def chat_completions(request: Request) -> StreamingResponse:
    await request.json()

    async def generate():
        for _ in range(5_000):
            chunk = {
                "id": "mock-deepseek-stall",
                "object": "chat.completion.chunk",
                "model": "deepseek-ai/DeepSeek-V4",
                "choices": [{
                    "index": 0,
                    "delta": {"reasoning_content": "思" * 16},
                    "finish_reason": None,
                }],
            }
            yield f"data: {json.dumps(chunk, ensure_ascii=False)}\n\n"
            await asyncio.sleep(0)

        while not await request.is_disconnected():
            await asyncio.sleep(0.1)

    return StreamingResponse(generate(), media_type="text/event-stream")

The harness used the real OpenAI SDK, AgentScope Agent, QwenPaw
OpenAIChatModelCompat, RetryChatModel, AgentExecutor, Envelope, SSE
serialization, and TaskTracker. Eight 2 MiB screenshot-shaped tool results
were accumulated in the same Envelope before the stalled model step.

For test speed, the heartbeat interval was reduced from 25 seconds to 0.25
seconds. The object serialized and retained on each tick was unchanged.
Twenty-four ticks correspond to ten minutes at the production interval.

macOS full-chain result Before After
Heartbeat SSE size 21.337 MiB < 1 KiB
RSS increase after 24 heartbeats 537.7 MiB 11.4 MiB
Stop completion at this load 0.019 s 0.001 s

This also shows that stop works when the process is healthy. At multi-GB
memory pressure, repeated large JSON serialization and allocation can make
both the UI and backend appear unresponsive before cancellation is handled.

2. Windows process-memory benchmark

The Windows benchmark uses the real QwenPaw schema and Envelope, six frozen
screenshot-shaped outputs at the 2 MiB raw-image limit, and the same SSE
strings retained by TaskTracker. It serializes 160 heartbeat events, equal
to 66 minutes 40 seconds at the production interval.

Reproduction script and successful windows-latest run:

Environment: Windows Server 2025, CPython 3.12.10.

160 heartbeat events Before After
SSE event size 16.0025 MiB ~0.0001 MiB
Retained SSE buffer 2.5004 GiB < 0.0001 GiB
Working Set delta 2560.6 MiB 0.0 MiB
Private Bytes delta 2565.7 MiB 0.0 MiB
Serialization time 2.090 s < 0.001 s

The issue is cross-platform Python retention, not a WebView leak. Windows
Task Manager reports qwenpaw-backend.exe and WebView2 as separate rows;
Windows was used here because it matches the reported environment and exposes
both Working Set and Private Bytes.

Evidence

The commit contains only the runtime fix and its regression tests. Benchmark
and mock-service files are intentionally not part of the PR.

PYTHONPATH=src:packages/qwenpawmail-mcp/src conda run -n QwenPaw pytest -q \
  tests/unit/runtime tests/unit/app/test_task_tracker.py \
  tests/unit/app/channels/test_console_channel.py \
  tests/unit/channels/test_console.py

169 passed in 21.82s
PYTHONPATH=src conda run -n QwenPaw pre-commit run --files \
  src/qwenpaw/runtime/envelope.py \
  tests/unit/runtime/test_envelope_heartbeat.py

All checks passed

The regression test verifies that a heartbeat does not contain a 1 MiB
accumulated output, stays below 256 bytes, and does not mutate the accumulated
response that is emitted normally at finalization.

Copilot AI lite review requested due to automatic review settings August 24, 2026 08:21
@github-project-automation github-project-automation Bot moved this to Todo in QwenPaw Aug 24, 2026
@github-actions

Copy link
Copy Markdown

Welcome to QwenPaw! 🐾

Hi @rayrayraykk, this is your 259th Pull Request.

📋 About PR Template

To help maintainers review your PR faster, please make sure to include:

  • Description - What this PR does and why
  • Type of Change - Bug fix / Feature / Breaking change / Documentation / Refactoring
  • Component(s) Affected - Core / Console / Channels / Skills / CLI / Documentation / Tests / CI/CD / Scripts
  • Checklist:
    • Run and pass pre-commit run --all-files
    • Run and pass relevant tests (pytest or as applicable)
    • Update documentation if needed
  • Testing - How to test these changes
  • Local Verification Evidence:
    pre-commit run --all-files
    # paste summary result
    
    pytest
    # paste summary result

Complete PR information helps speed up the review process. You can edit the PR description to add these details.

🙌 Join Developer Community

Thanks so much for your contribution! We'd love to invite you to join the official QwenPaw developer group! You can find the Discord and DingTalk group links under the "Developer Community" section on our docs page:
https://qwenpaw.agentscope.io/docs/community

We truly appreciate your enthusiasm—and look forward to your future contributions! 😊

We'll review your PR soon.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes unbounded backend memory growth during idle/stalled streaming by changing runtime heartbeat emissions to a lightweight keepalive event instead of re-serializing the full, cumulative AgentResponse on every tick. This prevents TaskTracker’s per-run SSE replay buffer from retaining gigabytes of duplicated response snapshots during long “Thinking” stalls.

Changes:

  • Updated Envelope.heartbeat() to emit a minimal Event(object="message", type="heartbeat") rather than the mutable accumulated _response.
  • Added regression tests to ensure heartbeats stay small and do not mutate or repeat accumulated turn output.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
src/qwenpaw/runtime/envelope.py Replaces cumulative full-response heartbeat snapshots with a lightweight heartbeat event to prevent SSE buffer memory blowups.
tests/unit/runtime/test_envelope_heartbeat.py Adds unit tests verifying heartbeat payload size and that accumulated response state is unchanged.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@rayrayraykk
rayrayraykk merged commit 504e811 into agentscope-ai:main Aug 24, 2026
27 of 28 checks passed
@github-project-automation github-project-automation Bot moved this from Todo to Done in QwenPaw Aug 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants