Contact Details
No response
What happened?
TestStandalone/SGLangCompound fails reproducibly (3/3 runs on a GKE L4 node, Qwen/Qwen2.5-0.5B, lmsysorg/sglang:v0.5.15):
standalone_test.go:83: inference changed after restore:
before=" Paris. It is the seat of the government of France. It is the"
after=" Paris. It is the largest city in Europe and the second largest in the"
The divergence is byte-identical across runs. All other standalone/k8s tests pass, including SGLangReleaseResume and both VLLM compound variants.
Root cause — not cuda-checkpoint, not the agent
The failure reproduces with the snapshot-agent entirely out of the loop, using only SGLang's own HTTP API (release_memory_occupation / resume_memory_occupation) on an engine launched with the suite's flags (--enable-memory-saver --enable-weights-cpu-backup --mem-fraction-static=0.5):
- On a fresh engine, repeated identical prompts (temperature 0) return identical completions, cold or radix-cache-warm.
-
- After one release/resume cycle, a radix-cache-hit request deterministically returns different logits than a cold request — flipping a borderline greedy token on a 0.5B model. POST /flush_cache restores the cold-path output; the next warm request diverges again. The remapped KV pool (torch_memory_saver VMM re-mapping) presumably changes the cached-KV decode kernel path numerics.
- SGLangCompound trips over this because it runs second on a shared engine: its "before" inference is a warm cache hit (the prompt was cached by SGLangReleaseResume's post-resume inference), while its "after" runs cold because release flushed the cache. SGLangReleaseResume passes since both of its inferences are cold.
Possible fixes
- Add --disable-radix-cache to the SGLang EngineSpec in tests/integration/snapshot-agent/engines.go — byte-equality checks can't span cache-flush boundaries, and the radix cache isn't under test. Verified: with this flag the full standalone suite passes.
- Alternative, keeping the cache enabled: POST /flush_cache before each baseline inference so before/after are both cold.
(written by claude, I do agree though with root cause)
Version
main
Steps to Reproduce
Run the tests
Environment
Relevant log output
Contact Details
No response
What happened?
TestStandalone/SGLangCompound fails reproducibly (3/3 runs on a GKE L4 node, Qwen/Qwen2.5-0.5B, lmsysorg/sglang:v0.5.15):
standalone_test.go:83: inference changed after restore:
before=" Paris. It is the seat of the government of France. It is the"
after=" Paris. It is the largest city in Europe and the second largest in the"
The divergence is byte-identical across runs. All other standalone/k8s tests pass, including SGLangReleaseResume and both VLLM compound variants.
Root cause — not cuda-checkpoint, not the agent
The failure reproduces with the snapshot-agent entirely out of the loop, using only SGLang's own HTTP API (release_memory_occupation / resume_memory_occupation) on an engine launched with the suite's flags (--enable-memory-saver --enable-weights-cpu-backup --mem-fraction-static=0.5):
Possible fixes
(written by claude, I do agree though with root cause)
Version
main
Steps to Reproduce
Run the tests
Environment
Relevant log output