[Bug]: Sleep-mode Level 2 wake does not reload speculative-decoding draft weights — silent ~2x decode slowdown, acceptance drops to 0

[Suppressor72]

[Bug]: Sleep-mode Level 2 wake does not reload speculative-decoding draft weights — silent ~2x decode slowdown, acceptance drops to 0

Summary

After waking a vLLM server from Level 2 sleep using the documented staged
sequence (wake_up(tags=["weights"]) → collective_rpc("reload_weights")
→ wake_up(tags=["kv_cache"]) → reset_prefix_cache), MTP speculative
decoding stops working entirely: every drafted token is rejected, and decode
throughput drops to roughly the no-speculation rate. Outputs remain
correct (verification rejects the garbage drafts), which makes the bug
silent — nothing crashes and no quality signal fires; you only notice via
throughput or the SpecDecoding metrics line.

Environment

• vLLM: 0.1.dev493+g4d9600003 (installed); bug also verified present on
  current main (fetched vllm/v1/worker/gpu_model_runner.py and
  vllm/v1/worker/gpu_worker.py on 2026-08-15 — reload_weights still
  iterates the main model's named_parameters() only, and wake_up still
  restores draft buffers only)
• GPUs: 2x RTX 5090 (SM120), TP=2
• Model: Qwen3.8-27B-FP8 (dense, FP8), --speculative-config '{"method":"mtp","num_speculative_tokens":2, "num_speculative_tokens_per_batch_size":[[1,2,2],[3,16,0]]}'
• Server flags: --enable-sleep-mode, VLLM_SERVER_DEV_MODE=1,
  kv-cache fp8, prefix caching on, custom all-reduce on
• PyTorch 2.13.0+cu130, CUDA 13.0

Reproduction (serving API)

# 1. cold boot with MTP spec decoding enabled; confirm baseline:
#    SpecDecoding metrics: Mean acceptance length: 2.6-2.8, ~110 tok/s decode

# 2. sleep level 2
curl -X POST 'http://localhost:8000/sleep?level=2'

# 3. documented staged wake
curl -X POST 'http://localhost:8000/wake_up?tags=["weights"]'
curl -X POST 'http://localhost:8000/collective_rpc' \
     -H 'Content-Type: application/json' -d '{"method":"reload_weights"}'
curl -X POST 'http://localhost:8000/wake_up?tags=["kv_cache"]'
curl -X POST 'http://localhost:8000/reset_prefix_cache'

# 4. generate; observe metrics:
#    SpecDecoding metrics: Mean acceptance length: 1.00, Accepted throughput: 0.00
#    decode ~55 tok/s (vs ~110 baseline)

Measured on our box (3-run medians, 256-token generations, ~1k-token prompt):
baseline 111.9 tok/s → post-wake 51.8 tok/s (46% of baseline). A cold
restart of the same config restores full speed, so it is not a config issue.

Root cause (source-level)

Level 2 sleep discards all weight memory — main model and drafter — and
saves only buffers to CPU (gpu_worker.sleep: _sleep_saved_buffers and
_sleep_saved_draft_buffers from named_buffers()). On wake,
gpu_worker.wake_up restores those buffers and re-reserves memory, but the
subsequent reload_weights call
(v1/worker/gpu_model_runner.py::reload_weights) builds its load set from
the main model's named_parameters():

model = self.get_model()
weights_to_load = {... for name, _ in model.named_parameters()}
...
loaded_weights = model.load_weights(weights_iterator)

get_draft_model() (→ drafter.model) is a separate module, so the MTP
drafter's parameters are never written after the L2 discard. The drafter
runs on reserved-but-uninitialized memory → acceptance 0 → the verifier
rejects everything → correct outputs at no-spec speed.

The buffer save/restore path already treats main and draft models
symmetrically (wake_up lines restoring _sleep_saved_draft_buffers); the
parameter reload path simply lacks the symmetric handling.

Impact

• Silent ~2x decode slowdown after every L2 wake with MTP/EAGLE-style
  speculative decoding (any backend whose drafter is a separate module).
• Model-switch setups (the documented sleep-mode use case) hit this on the
  first swap and never recover without a restart.

Related

• [Bug]: sleep level 2 causes gibberish outputs #29341 (L2 wake produces gibberish main-model outputs — same
  "L2 restore incomplete" family, different layer)
• [Bug]: After wake up from level 2 sleep, model cannot load weights properly #16564 (L2 wake weight load failures)
• [Bug]: reload_weights (via collective_rpc) Fails to Load Quantized Models Correctly, Causing Tensor Shape AssertionError #28606 (reload_weights issues with quantized formats)
• Runtime Draft Weight Update for Speculative Decoding #46725 (open PR: Runtime Draft Weight Update for Speculative Decoding —
  adjacent machinery)
• [RFC]: Better support for weight updating while waking up from sleep mode for RLHF #15254 / [RFC] [QeRL]: Online Quantization and Model Reloading #30359 (RFCs on wake-up weight-update design)

Proposed direction

Extend the L2 wake reload path so drafter parameters are reloaded from the
same checkpoint stream (or via the drafter's own load path) before KV-cache
wake. Happy to submit a PR — pointers on preferred shape (extend
reload_weights vs. a drafter-specific hook) welcome.

---

Bug found and reported by an AI agent (Hermes/Aether, operated by Greg
Weyer) during serving-infrastructure debugging; reproduction and source
analysis verified against the installed build.