fix(qwen36): restore MoE device-tilemap (MOE_GPU) correctness — masked down-dequant no-op#602
Closed
James-CUDA wants to merge 1 commit into
Closed
Conversation
…d down dequant no-op SPARKINFER_PREFILL_MOE_GPU=1 (device tilemap, skips the per-layer D2H counts sync) is silently broken on main: the down projection's masked Q->int8 dequant has cols=mffn=512, which the vectorized fast-mask kernel declines (cols not a multiple of 1024), and launch_gguf_dequant_rows_i8_mask ignored that false return — so the int8 down weights were never written and the down GEMM ran on stale/garbage data. Batched-vs-token-loop prefill_check collapses to TOP1 ~0.375 / KL ~0.35 @512 (this is the gittensor-ai-lab#586 corruption that forced the path default-off). Fix: launch_gguf_dequant_rows_i8_mask now falls back to a correct masked slow kernel (single-pass, register-resident, same dequant/amax/round as the host deq_rows_i8_kernel; dead experts exit via the counts mask) when the fast path declines. Restores prefill_check to TOP1 0.94 / KL ~0.01 @512 with MOE_GPU=1. Keeps SPARKINFER_PREFILL_MOE_GPU default OFF: the device path is not currently a prefill speedup (it must dequant all experts per group rather than the host path's live-only coalesced runs), so this is a correctness fix for the opt-in flag, not a perf change. The default host-tilemap path is unaffected.
Closed — RTX 5090 checkbox not tickedThis PR was auto-closed because the template includes Tested on RTX 5090 as To submit for review:
If this PR does not need GPU eval (e.g. docs-only), remove the proof-of-speedup section from the description instead of leaving an unchecked box. Automated by rtx5090-required CI. Maintainers and |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Correctness fix (not a speed change).
SPARKINFER_PREFILL_MOE_GPU=1— the Qwen3.6 device-tilemap prefill path that skips the per-layer D2H counts sync — is silently broken onmain: it produces garbage output.Root cause: the MoE down projection's masked
Q→int8dequant runs withcols = moe_ffn = 512, which the vectorized fast-mask kernel declines (it only acceptscolsthat are a multiple of 1024).launch_gguf_dequant_rows_i8_maskignored thatfalsereturn, so the int8 down weights were never written and the down GEMM ran on stale/garbage data. This is the #586 corruption that forced the path default-off.Fix:
launch_gguf_dequant_rows_i8_masknow falls back to a correct masked slow kernel (single-pass, register-resident, identical dequant/amax/round to the hostdeq_rows_i8_kernel; dead experts exit via the counts mask) when the fast path declines. The default host-tilemap path is untouched, andSPARKINFER_PREFILL_MOE_GPUstays default OFF (the device path is not currently a prefill speedup — it must dequant all experts per group rather than the host path's live-only coalesced runs — so this fixes the opt-in flag rather than changing perf).Proof (correctness, not speed)
Not a speedup PR — no
bench.shbefore/after and the RTX-5090 speedup box is intentionally left unticked. Evidence is the prefill fidelity gate, on RTX 5090 / Qwen3.6-35B-A3B-UD-Q4_K_M,qwen3_gguf_prefill_check <model> 512 16withSPARKINFER_PREFILL_MOE_GPU=1:The host-tilemap default path (
MOE_GPU=0) is unchanged (TOP1 1.0 @512, byte-identical to before).sm_120) — correctness only; no speed claim