Add mxfp8 support for online quantization, Triton dense linear, and CUTLASS MoE

[zianglih]

Motivation

@HumansAnd

#17093
This PR adds mxfp8 quantization support to SGLang, using Triton for dense linear layer, and existing mxfp8 CUTLASS groped GEMM kernel in sgl-kernel from #13731 for MoE.

Online mxfp8 quantization from bf16 checkpoints and serving mxfp8 checkpoints directly are both supported.

Modifications

Accuracy Tests

# Eval:
python3 benchmark/gsm8k/bench_sglang.py --num-shots 8 --num-questions 1209 --parallel 1209 --platinum

# bf16:
python -m sglang.launch_server --tp 2 --model Qwen/Qwen3-30B-A3B-Instruct-2507 --fp8-gemm-backend triton --moe-runner-backend cutlass  &
# Trial 1
Accuracy: 0.964
Invalid: 0.000
Latency: 14.543 s
Output throughput: 11779.580 token/s
# Trial 2
Accuracy: 0.964
Invalid: 0.000
Latency: 13.427 s
Output throughput: 12778.580 token/s
# Trial 3
Accuracy: 0.964
Invalid: 0.000
Latency: 13.049 s
Output throughput: 13128.875 token/s
# Online mxfp8:
python -m sglang.launch_server --tp 2 --model Qwen/Qwen3-30B-A3B-Instruct-2507 --quantization mxfp8 --fp8-gemm-backend triton --moe-runner-backend cutlass  &
# Trial 1
Accuracy: 0.964
Invalid: 0.000
Latency: 23.825 s
Output throughput: 7197.115 token/s
# Trial 2
Accuracy: 0.966
Invalid: 0.000
Latency: 18.390 s
Output throughput: 9310.458 token/s
# Trial 3
Accuracy: 0.966
Invalid: 0.000
Latency: 18.178 s
Output throughput: 9419.016 token/s
# Offline mxfp8:
python -m sglang.launch_server --tp 1 --model /data/models/Qwen3-30B-A3B-Instruct-2507-MXFP8 --fp8-gemm-backend triton --moe-runner-backend cutlass  &
# Trial 1
Accuracy: 0.963
Invalid: 0.000
Latency: 18.698 s
Output throughput: 9075.207 token/s
# Trial 2
Accuracy: 0.963
Invalid: 0.000
Latency: 18.297 s
Output throughput: 9273.994 token/s
# Trial 3
Accuracy: 0.963
Invalid: 0.000
Latency: 17.752 s
Output throughput: 9558.566 token/s

Benchmarking and Profiling

Next Steps

• Latest DeepGEMM includes mxfp8 & mxfp4 kernels by passing in recipe = (1, 1, 32). Add DeepGEMM backend for mxfp8 when sgl-kernel version is bumped.
  • Some existing DeepGEMM entrypoint refactoring work: Expand deep_gemm entrypoint to support more FP8 recipes. #17294
• Improve Triton mxfp8 kernel performance.
• Blackwell mxfp8 RL integration.

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

Review Process

1. Ping Merge Oncalls to start the PR flow. See the PR Merge Process.
2. Get approvals from CODEOWNERS and other reviewers.
3. Trigger CI tests with comments or contact authorized users to do so.
   • /tag-run-ci-label, /rerun-failed-ci, /tag-and-rerun-ci
4. After green CI and required approvals, ask Merge Oncalls to merge.

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

[ispobock]

/tag-and-rerun-ci

[zianglih]

Implemented a few minor fixes for /update_weights_from_disk.
Test for online mxfp8 quantization:

python -m sglang.launch_server --tp 2 --model Qwen/Qwen3-30B-A3B-Instruct-2507 --quantization mxfp8 --fp8-gemm-backend triton --moe-runner-backend cutlass  &
python3 benchmark/gsm8k/bench_sglang.py --num-shots 8 --num-questions 1209 --parallel 1209 --platinum
Accuracy: 0.964
Invalid: 0.000
Latency: 20.583 s
Output throughput: 8330.881 token/s

Test for Qwen3-4B-Instruct-2507-MXFP8 (dense), with /update_weights_from_disk:

python -m sglang.launch_server --tp 1 --model /data/models/Qwen3-4B-Instruct-2507-MXFP8 --fp8-gemm-backend triton --moe-runner-backend cutlass  &
python3 benchmark/gsm8k/bench_sglang.py --num-shots 8 --num-questions 1209 --parallel 1209 --platinum
Accuracy: 0.849
Invalid: 0.000
Latency: 16.115 s
Output throughput: 13397.779 token/s
curl -sS http://localhost:30000/update_weights_from_disk \
  -H 'Content-Type: application/json' \
  -d '{
    "model_path": "/data/models/Qwen3-4B-Instruct-2507-MXFP8",
    "flush_cache": true,
    "abort_all_requests": false
  }'
python3 benchmark/gsm8k/bench_sglang.py --num-shots 8 --num-questions 1209 --parallel 1209 --platinum
Accuracy: 0.849
Invalid: 0.000
Latency: 15.149 s
Output throughput: 14251.889 token/s

Test for Qwen3-30B-A3B-Instruct-2507-MXFP8 (MoE), with /update_weights_from_disk:

python -m sglang.launch_server --tp 1 --model /data/models/Qwen3-30B-A3B-Instruct-2507-MXFP8 --fp8-gemm-backend triton --moe-runner-backend cutlass  &
python3 benchmark/gsm8k/bench_sglang.py --num-shots 8 --num-questions 1209 --parallel 1209 --platinum
Accuracy: 0.963
Invalid: 0.000
Latency: 18.686 s
Output throughput: 9081.105 token/s
curl -sS http://localhost:30000/update_weights_from_disk \
  -H 'Content-Type: application/json' \
  -d '{
    "model_path": "/data/models/Qwen3-30B-A3B-Instruct-2507-MXFP8",
    "flush_cache": true,
    "abort_all_requests": false
  }'
python3 benchmark/gsm8k/bench_sglang.py --num-shots 8 --num-questions 1209 --parallel 1209 --platinum
Accuracy: 0.963
Invalid: 0.000
Latency: 19.230 s
Output throughput: 8823.926 token/s

Implementation is now fully complete.

[fxmarty-amd]

This is surprising given the snippet below?

sglang/python/sglang/srt/layers/quantization/fp8.py

Lines 409 to 412 in 0769de9

	elif self.use_mxfp8:
	if not self.is_checkpoint_fp8_serialized:
	self._quantize_mxfp8_weights(layer)
	return

Should this rather say that it is simply untested? Or should this error be removed?

[zianglih]

mxfp8 online quantization from bf16 is tested as in the Accuracy Test section. This raise is never triggered so can be removed.