馃殌 The feature, motivation and pitch
vLLM can load OCP MXFP6 checkpoints through Quark, but CUDA execution currently uses software emulation. PR #21166 added the format and emulation support and left native OCP MX kernels as follow-up work.
We built mxfp6-sm120, a BSD-licensed PyTorch CUDA extension for native OCP W6A8 execution on NVIDIA SM120. It keeps weights in packed MXFP6 E3M2 with UE8M0 scales, quantizes FP16/BF16 activations to MXFP8 E4M3 at runtime, and accumulates in FP32. The current implementation includes:
- a general Dense GEMM operator;
- a Qwen3.5 routed-MoE path covering routing, W1, activation processing, W2, shared-expert combine, and final reduction;
- persistent workspaces and CUDA Graph replay for the tested execution paths.
A public vLLM v0.25.1 reproducer loads and serves Qwen3.5-27B and Qwen3.5-35B-A3B on two RTX 5090 GPUs with tensor parallel size 2. Compared with the corresponding official FP8 checkpoints, MXFP6 improved output throughput by 9.11% for Qwen3.5-27B and 16.95% for Qwen3.5-35B-A3B.
This issue documents the working implementation and measured scope. If this evidence is not yet sufficient to evaluate an upstream contribution, we can add targeted correctness coverage, workloads, hardware configurations, or fallback tests.
Alternatives
The existing Quark software-emulation path remains a functional fallback, but it does not execute the packed weights with native SM120 MX instructions. The repository also contains a narrowly versioned vLLM compatibility patch. That patch demonstrates end-to-end integration for the two tested models; it is not proposed as an upstream patch.
Additional context
Public end-to-end results
The public reproduction uses vLLM v0.25.1, two PIX-connected RTX 5090 GPUs, tensor parallel size 2, and four fresh service lifecycles in FP8/MXFP6/MXFP6/FP8 order.
| Model and workload |
FP8 output tok/s |
MXFP6 output tok/s |
Gain |
| Qwen3.5-27B, 3000 input / 1000 output, c32 |
1229.90 |
1341.97 |
+9.11% |
Qwen3.5-35B-A3B, public random-mm, c4 |
678.37 |
793.33 |
+16.95% |
Standalone measurements report a 1.633x geometric-mean speedup over vLLM block-FP8 for the five measured Qwen3.5-27B Dense shapes, and 1.278x over the vLLM FP8 MoE path for the complete Qwen3.5-35B-A3B TP2 MoE layer at B16-B96. Operator, layer, and service results have different measurement boundaries and are not additive.
Internal runtime reference
We also measured the package with an internally optimized vLLM 0.25.1 stack: 22.44% higher output throughput for Qwen3.5-27B and 13.58% for Qwen3.5-35B-A3B. On the 35B-A3B workload, 742 greedy reference cases changed full-character similarity by -0.57 percentage points and answer-section similarity by -0.32 points relative to FP8. Neither paired 95% interval detected a significant loss. This is reference-output fidelity, not task accuracy, and the internal serving results are not claims about the public reproducer.
Reproduction and evidence
馃殌 The feature, motivation and pitch
vLLM can load OCP MXFP6 checkpoints through Quark, but CUDA execution currently uses software emulation. PR #21166 added the format and emulation support and left native OCP MX kernels as follow-up work.
We built mxfp6-sm120, a BSD-licensed PyTorch CUDA extension for native OCP W6A8 execution on NVIDIA SM120. It keeps weights in packed MXFP6 E3M2 with UE8M0 scales, quantizes FP16/BF16 activations to MXFP8 E4M3 at runtime, and accumulates in FP32. The current implementation includes:
A public vLLM v0.25.1 reproducer loads and serves Qwen3.5-27B and Qwen3.5-35B-A3B on two RTX 5090 GPUs with tensor parallel size 2. Compared with the corresponding official FP8 checkpoints, MXFP6 improved output throughput by 9.11% for Qwen3.5-27B and 16.95% for Qwen3.5-35B-A3B.
This issue documents the working implementation and measured scope. If this evidence is not yet sufficient to evaluate an upstream contribution, we can add targeted correctness coverage, workloads, hardware configurations, or fallback tests.
Alternatives
The existing Quark software-emulation path remains a functional fallback, but it does not execute the packed weights with native SM120 MX instructions. The repository also contains a narrowly versioned vLLM compatibility patch. That patch demonstrates end-to-end integration for the two tested models; it is not proposed as an upstream patch.
Additional context
Public end-to-end results
The public reproduction uses vLLM v0.25.1, two PIX-connected RTX 5090 GPUs, tensor parallel size 2, and four fresh service lifecycles in FP8/MXFP6/MXFP6/FP8 order.
random-mm, c4Standalone measurements report a 1.633x geometric-mean speedup over vLLM block-FP8 for the five measured Qwen3.5-27B Dense shapes, and 1.278x over the vLLM FP8 MoE path for the complete Qwen3.5-35B-A3B TP2 MoE layer at B16-B96. Operator, layer, and service results have different measurement boundaries and are not additive.
Internal runtime reference
We also measured the package with an internally optimized vLLM 0.25.1 stack: 22.44% higher output throughput for Qwen3.5-27B and 13.58% for Qwen3.5-35B-A3B. On the 35B-A3B workload, 742 greedy reference cases changed full-character similarity by -0.57 percentage points and answer-section similarity by -0.32 points relative to FP8. Neither paired 95% interval detected a significant loss. This is reference-output fidelity, not task accuracy, and the internal serving results are not claims about the public reproducer.
Reproduction and evidence