Skip to content

fix(mlnode): scheduler-heartbeat liveness for vLLM runner + damped proxy health probe#1421

Open
baychak wants to merge 1 commit into
gonka-ai:mainfrom
kaitakuai:fix/vllm-runner-heartbeat-liveness
Open

fix(mlnode): scheduler-heartbeat liveness for vLLM runner + damped proxy health probe#1421
baychak wants to merge 1 commit into
gonka-ai:mainfrom
kaitakuai:fix/vllm-runner-heartbeat-liveness

Conversation

@baychak

@baychak baychak commented Jul 7, 2026

Copy link
Copy Markdown

Problem

vLLM /health is the wrong liveness signal in both directions, and mlnode relies on it twice:

  1. Real hang, /health 200. A TP worker deadlocks on the shm collective: scheduler frozen, zero throughput — but AsyncLLM.check_health only reads the errored flag. The node silently serves nothing until someone notices (5–22 min in production).
  2. Busy engine, /health timeout. A saturated engine misses the 2 s deadline: runner.is_available() reports dead → a healthy node gets restarted (observed 8+/day); the proxy prober flips on a single miss → the compatibility server stops, dropping in-flight relays (23 flaps/2.3 h).

Fix

  • runner.py: liveness = vllm:iteration_tokens_total_count heartbeat — advances on every engine step, freezes only on a stall. Per-instance; re-baselines after engine restarts; aggregates multi-registry scrapes; a failed scrape can't pass for idle; ConnectTimeout = saturation (grace path), refused = dead. MLNODE_HANG_GRACE_SEC (default 120, 0 disables).
  • proxy.py: prober timeout 2 → 10 s; unhealthy only after 3 consecutive failures, instant recovery.

Validation

21 unit tests. 4×B300 Kimi-K2.6 fleet, several days: false restarts → 0 (incl. a full hour at max_num_seqs saturation), real stalls detected in ~2 min and self-healed via the existing restart path, proxy flaps → 0. Detection layer only — the underlying vLLM stall is addressed in gonka-ai/vllm#58.

@baychak
baychak force-pushed the fix/vllm-runner-heartbeat-liveness branch 3 times, most recently from 3e0c61b to 1ebd975 Compare July 10, 2026 18:26
vLLM /health is wrong in both directions: it returns 200 during a
scheduler deadlock (check_health only reads the `errored` flag) and
misses its deadline while merely busy -- so healthy nodes got restarted
under load while real hangs served nothing silently.

runner.py: key is_available() off vllm:iteration_tokens_total_count,
which advances on every engine step and freezes only on a stall.
Split as _scrape_heartbeat (per-port /metrics read; ConnectTimeout is
saturation, refused is dead) + _heartbeat_verdict (per-port state
machine: re-baselines across engine restarts, aggregates multi-registry
scrapes, a failed scrape cannot pass for idle, three consecutive frozen
verdicts before unhealthy). MLNODE_HANG_GRACE_SEC tunes the window
(default 120, 0 disables).

proxy.py: damp the backend prober -- timeout 2s -> 10s, unhealthy only
after 3 consecutive failures, instant recovery.

21 unit tests. Validated on a 4xB300 Kimi-K2.6 fleet: false restarts
-> 0 (incl. 1h at full saturation), real stalls detected in ~2 min,
proxy flap cycles -> 0.
@baychak
baychak force-pushed the fix/vllm-runner-heartbeat-liveness branch from 1ebd975 to e4a85c3 Compare July 10, 2026 19:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant