fix(mlnode): scheduler-heartbeat liveness for vLLM runner + damped proxy health probe#1421
Open
baychak wants to merge 1 commit into
Open
fix(mlnode): scheduler-heartbeat liveness for vLLM runner + damped proxy health probe#1421baychak wants to merge 1 commit into
baychak wants to merge 1 commit into
Conversation
baychak
force-pushed
the
fix/vllm-runner-heartbeat-liveness
branch
3 times, most recently
from
July 10, 2026 18:26
3e0c61b to
1ebd975
Compare
vLLM /health is wrong in both directions: it returns 200 during a scheduler deadlock (check_health only reads the `errored` flag) and misses its deadline while merely busy -- so healthy nodes got restarted under load while real hangs served nothing silently. runner.py: key is_available() off vllm:iteration_tokens_total_count, which advances on every engine step and freezes only on a stall. Split as _scrape_heartbeat (per-port /metrics read; ConnectTimeout is saturation, refused is dead) + _heartbeat_verdict (per-port state machine: re-baselines across engine restarts, aggregates multi-registry scrapes, a failed scrape cannot pass for idle, three consecutive frozen verdicts before unhealthy). MLNODE_HANG_GRACE_SEC tunes the window (default 120, 0 disables). proxy.py: damp the backend prober -- timeout 2s -> 10s, unhealthy only after 3 consecutive failures, instant recovery. 21 unit tests. Validated on a 4xB300 Kimi-K2.6 fleet: false restarts -> 0 (incl. 1h at full saturation), real stalls detected in ~2 min, proxy flap cycles -> 0.
baychak
force-pushed
the
fix/vllm-runner-heartbeat-liveness
branch
from
July 10, 2026 19:45
1ebd975 to
e4a85c3
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
vLLM
/healthis the wrong liveness signal in both directions, and mlnode relies on it twice:/health200. A TP worker deadlocks on the shm collective: scheduler frozen, zero throughput — butAsyncLLM.check_healthonly reads theerroredflag. The node silently serves nothing until someone notices (5–22 min in production)./healthtimeout. A saturated engine misses the 2 s deadline:runner.is_available()reports dead → a healthy node gets restarted (observed 8+/day); the proxy prober flips on a single miss → the compatibility server stops, dropping in-flight relays (23 flaps/2.3 h).Fix
runner.py: liveness =vllm:iteration_tokens_total_countheartbeat — advances on every engine step, freezes only on a stall. Per-instance; re-baselines after engine restarts; aggregates multi-registry scrapes; a failed scrape can't pass for idle;ConnectTimeout= saturation (grace path), refused = dead.MLNODE_HANG_GRACE_SEC(default 120,0disables).proxy.py: prober timeout 2 → 10 s; unhealthy only after 3 consecutive failures, instant recovery.Validation
21 unit tests. 4×B300 Kimi-K2.6 fleet, several days: false restarts → 0 (incl. a full hour at max_num_seqs saturation), real stalls detected in ~2 min and self-healed via the existing restart path, proxy flaps → 0. Detection layer only — the underlying vLLM stall is addressed in gonka-ai/vllm#58.