ROCm · coketaste · Jul 28, 2026 · Jul 28, 2026
@@ -14,7 +14,7 @@ Below are blueprints of supported models along with their documentation.
 | [**xDiT diffusion inference**](benchmark/xdit/README.md) | Diffusion Transformer inference using xDiT | FLUX.1, FLUX.1 Kontext, FLUX.2, FLUX.2 Klein, HunyuanVideo, HunyuanVideo 1.5, LTX-2, Stable Diffusion 3.5, Wan 2.1, Wan 2.2, Z-Image Turbo |
 | [**JAX MaxText training**](benchmark/jax_maxtext/README.md) | Train LLMs on AMD Instinct GPUs using JAX MaxText | Llama 2 7B/70B, Llama 3/3.1 8B/70B, Llama 3.1 405B, Llama 3.3 70B, DeepSeek-V2-lite 16B, Mixtral-8x7B |
 | [**vLLM inference**](benchmark/vllm/README.md) | LLM Inference with vLLM on AMD Instinct GPUs | DeepSeek-R1, gpt-oss-20b/120b, Llama-2-70b, Llama-3.1-8b/405b, Llama-3.3-70b, Llama-4-Scout/Maverick, Mixtral-8x7b/8x22b, Phi-4, Qwen3-8b/32b/30b-a3b/235b-a22b |
-| [**SGLang inference**](benchmark/sglang/README.md) | LLM Inference with SGLang on AMD Instinct GPUs | DeepSeek-R1-Distill-Qwen-32B |
+| [**SGLang inference**](benchmark/sglang/README.md) | LLM Inference with SGLang on AMD Instinct GPUs | DeepSeek-R1-Distill-Qwen-32B, Kimi-K3 |
 | [**PyTorch training**](benchmark/pytorch_train/README.md) | Train LLMs on AMD Instinct GPUs using AMD's Primus | Llama 2/3/3.1/3.2/3.3/4, GPT-OSS 20B/120B, Qwen2/2.5/3, Flux, SDXL, DLRM, and others |
 | [**PyTorch inference**](benchmark/pytorch_inference/README.md) | Inference recipes for Multimodal, video and vision transformer models | Mochi video, Chai-1, CLIP (ViT-B-32), Wan2.1, Janus-Pro-7B, HunyuanVideo |
 | [**Megatron-LM training**](benchmark/megatron_lm/README.md) | Train LLMs on AMD Instinct GPUs using ROCm Megatron-LM | Llama 2 7B/70B, Llama 3/3.1 8B/70B, Llama 3.3 70B, DeepSeek-V2-lite, DeepSeek-V3, Mixtral 8x7B/8x22B, Qwen 2.5 7B/72B |

@@ -80,6 +80,48 @@ users can also change the benchmarking parameters. Refer to the [Standalone benc
 | model_name                              |
 | --------------------------------------- |
 | pyt_sglang_deepseek-r1-distill-qwen-32b  |
+| pyt_sglang_kimi-k3                       |
+| pyt_sglang_kimi-k3_dspark                |
+
+>[!NOTE]
+>The two `pyt_sglang_kimi-k3*` entries are the exception to everything described above. They track the
+>AMD day-0 recipes in [sgl-project/sglang#32548](https://github.com/sgl-project/sglang/issues/32548)
+>(day-0 support: [#32541](https://github.com/sgl-project/sglang/pull/32541), see also the
+>[SGLang K3 cookbook](https://docs.sglang.io/cookbook/autoregressive/Moonshotai/Kimi-K3)) and differ in
+>four ways:
+>
+>- **Image.** They build from `lmsysorg/sglang-rocm:rocm720-mi35x-k3-20260727` via
+>  [docker/pyt_sglang_kimi_k3.ubuntu.amd.Dockerfile](../../docker/pyt_sglang_kimi_k3.ubuntu.amd.Dockerfile),
+>  not the shared `lmsysorg/sglang:v0.4.5-rocm630` above, which predates K3 support.
+>- **Benchmark.** They measure *online serving* (`sglang serve` + `sglang.benchmark.serving`) through
+>  [scripts/sglang/run_sglang.py](../../scripts/sglang/run_sglang.py) and
+>  [scripts/sglang/configs/kimi_k3.yaml](../../scripts/sglang/configs/kimi_k3.yaml), rather than the
+>  offline latency/throughput path documented below.
+>- **Hardware.** 8x MI350X/MI355X (gfx950) TP8 only, hence `skip_gpu_arch: gfx942`. The checkpoint is
+>  large, so make sure `HF_HUB_CACHE` has room; `pyt_sglang_kimi-k3_dspark` additionally pulls the
+>  [RadixArk/Kimi-K3-DSpark](https://huggingface.co/RadixArk/Kimi-K3-DSpark) draft checkpoint.
+>- **Invocation.** They carry no sweep tag, so a tag run does not pull the checkpoint and hold 8 GPUs.
+>  Run them explicitly by name:
+>
+>```sh
+>madengine run --tags pyt_sglang_kimi-k3 --keep-model-dir --live-output
+>madengine run --tags pyt_sglang_kimi-k3_dspark --keep-model-dir --live-output
+>```
+>
+>The sweep is 8192-token input / 1024-token output at concurrency 2/4/8/16/32. The issue does not state
+>its input and output lengths; they were recovered from its own tables, where
+>`(E2EL - TTFT) / TPOT + 1` lands on ~1024 output tokens on every row and
+>`concurrency x (inp + out) / E2EL` reproduces the reported total throughput only at 8192 input tokens.
+>Keeping that shape is what makes MAD's numbers comparable to the issue's.
+>
+>The runner emits `--tp-size` where the issue writes `--tp`. There is no `--tp` server argument;
+>`tp_size` has a single alias, `--tensor-parallel-size`, and `--tp` resolves only through argparse
+>prefix matching, which a future `--tp*` option would silently break.
+
+>[!WARNING]
+>The published performance tables were measured on **MI355X**. A commenter on the tracking issue reports
+>much weaker results on **MI350X** with untuned AITER kernels. Both report as gfx950, so `skip_gpu_arch`
+>cannot distinguish them — treat MI350X numbers from this recipe as unvalidated.
 
 ### Standalone benchmarking              
 -----------------------------

@@ -0,0 +1,50 @@
+# CONTEXT {'gpu_vendor': 'AMD', 'guest_os': 'UBUNTU'}
+###############################################################################
+#
+# MIT License
+#
+# Copyright (c) 2025 Advanced Micro Devices, Inc.
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all
+# copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+#
+#################################################################################
+# Kimi K3 day-0 support landed in sgl-project/sglang#32541 and is not in a
+# tagged SGLang ROCm release yet; this model-specific image is the only ROCm
+# build carrying the KDA / Stable LatentMoE / AITER A8W4 support the checkpoint
+# needs. Kept separate from docker/pyt_sglang, which is still on the v0.4.5
+# rocm630 base that the existing SGLang entry is validated against.
+#
+# The tag is the day-0 image named in the AMD tracking issue
+# https://github.com/sgl-project/sglang/issues/32548, which is what its MI355X
+# performance tables were measured against. A newer rocm720-mi35x-k3-20260728
+# tag exists on Docker Hub but nothing published ties it to the recipe, so it is
+# deliberately not adopted here.
+#
+# Fold this back into docker/pyt_sglang once K3 lands in a versioned ROCm image.
+ARG BASE_DOCKER=lmsysorg/sglang-rocm:rocm720-mi35x-k3-20260727
+
+FROM $BASE_DOCKER
+
+USER root
+ENV WORKSPACE_DIR=/workspace
+RUN mkdir -p $WORKSPACE_DIR
+WORKDIR $WORKSPACE_DIR
+
+# record configuration for posterity
+RUN pip3 list
@@ -2194,6 +2194,46 @@
     "args":
      "--model_repo deepseek-ai/DeepSeek-R1-Distill-Qwen-32B --test_option latency --num_gpu 8 --datatype bfloat16 --dataset random --batch_size 1,8,32 --lat_input_output_len '128:128;128:1024;1024:128;1024:1024'"
   },
+  {
+    "name": "pyt_sglang_kimi-k3",
+    "url": "",
+    "dockerfile": "docker/pyt_sglang_kimi_k3",
+    "scripts": "scripts/sglang/run_kimi_k3.sh",
+    "data": "huggingface",
+    "n_gpus": "-1",
+    "owner": "mad.support@amd.com",
+    "training_precision": "",
+    "multiple_results": "perf_Kimi-K3.csv",
+    "tags": [
+      "pyt",
+      "sglang",
+      "inference"
+    ],
+    "timeout": -1,
+    "skip_gpu_arch": "gfx942",
+    "args":
+     "--model_repo moonshotai/Kimi-K3 --config configs/kimi_k3.yaml --variant nospec"
+  },
+  {
+    "name": "pyt_sglang_kimi-k3_dspark",
+    "url": "",
+    "dockerfile": "docker/pyt_sglang_kimi_k3",
+    "scripts": "scripts/sglang/run_kimi_k3.sh",
+    "data": "huggingface",
+    "n_gpus": "-1",
+    "owner": "mad.support@amd.com",
+    "training_precision": "",
+    "multiple_results": "perf_Kimi-K3.csv",
+    "tags": [
+      "pyt",
+      "sglang",
+      "inference"
+    ],
+    "timeout": -1,
+    "skip_gpu_arch": "gfx942",
+    "args":
+     "--model_repo moonshotai/Kimi-K3 --config configs/kimi_k3.yaml --variant dspark"
+  },
   {
     "name": "pyt_hy_video",
     "url": "",

@@ -0,0 +1,90 @@
+#################################################################################
+#
+# MIT License
+#
+# Copyright (c) 2025 Advanced Micro Devices, Inc.
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all
+# copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+#
+#################################################################################
+#
+# Kimi K3 on MI355X, tracking the AMD day-0 recipes in
+# https://github.com/sgl-project/sglang/issues/32548 (day-0 support: #32541).
+#
+# Two variants, selected with --variant because both serve the same model:
+#   nospec  the non speculative-decoding recipe
+#   dspark  the same server plus the DSpark draft checkpoint
+#
+# Sweep shape. The issue's tables do not state input/output lengths. They are
+# recovered from the tables themselves: (E2EL - TTFT) / TPOT + 1 lands on ~1024
+# output tokens for every row, and concurrency * (inp + out) / E2EL reproduces
+# the reported total throughput only at inp=8192 (e.g. concurrency 8:
+# 8 * 9216 / 31.297 s = 2355.8 vs 2356.21 reported). MAD's usual 1024/1024 would
+# produce numbers that cannot be compared against the tracking issue.
+#
+# max_concurrency mirrors the issue's rows exactly (2/4/8/16/32), so each variant
+# expands into the same five measurements the tables report.
+#
+# --disable-radix-cache comes from the recipe itself, not from MAD benchmark
+# hygiene, and so is spelled out in extra_args rather than forced by the runner.
+#
+# The runner emits --tp-size, not the --tp shown in the issue: ServerArgs
+# declares tp_size with a single alias, --tensor-parallel-size, and there is no
+# --tp option. "--tp 8" only parses through argparse prefix matching, which any
+# future --tp* option would silently break.
+
+- benchmark: serving
+  variant: nospec
+  model: moonshotai/Kimi-K3
+  tp: 8
+  inp: 8192
+  out: 1024
+  dtype: bfloat16
+  max_concurrency: 2 4 8 16 32
+  env: &k3_env
+    SGLANG_USE_AITER: 1
+    SGLANG_AITER_K3_OPT: 1
+    AITER_FLYDSL_FORCE: 1
+    # selects the AITER A8W4 MoE path for the natively-MXFP4 K3 weights
+    AITER_SITUV2_A8W4: 1
+  extra_args: &k3_extra_args
+    --attention-backend: triton
+    --mem-fraction-static: 0.85
+    --cuda-graph-max-bs-decode: 256
+    --disable-radix-cache: true
+    # K3 always thinks; these split reasoning and tool calls out of the answer
+    --reasoning-parser: kimi_k3
+    --tool-call-parser: kimi_k3
+
+# DSpark reuses the anchors above so the shared recipe has one definition. It is
+# reported separately in the issue because it also yields an accept-length
+# column, which the runner records when the server reports one.
+- benchmark: serving
+  variant: dspark
+  model: moonshotai/Kimi-K3
+  tp: 8
+  inp: 8192
+  out: 1024
+  dtype: bfloat16
+  max_concurrency: 2 4 8 16 32
+  env: *k3_env
+  extra_args:
+    <<: *k3_extra_args
+    --speculative-draft-model-path: RadixArk/Kimi-K3-DSpark
+    --speculative-algorithm: DSPARK
@@ -0,0 +1,64 @@
+#!/bin/bash
+###############################################################################
+#
+# MIT License
+#
+# Copyright (c) 2025 Advanced Micro Devices, Inc.
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all
+# copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+#
+#################################################################################
+# Entry point for the Kimi K3 SGLang serving benchmark. Separate from run.sh,
+# which drives the offline bench_one_batch / bench_offline_throughput path and
+# is gated to gfx94x; K3 is an online serving recipe on gfx950.
+set -ex
+
+# Preliminary setup
+if [[ -z "${HF_HUB_CACHE:-}" ]]; then
+    export HF_HUB_CACHE="/myworkspace"
+fi
+export HF_TOKEN=$MAD_SECRETS_HFTOKEN
+
+while [[ "$#" -gt 0 ]]; do
+    case $1 in
+        --model_repo) MODEL="$2"; shift ;;
+        --config) CONFIG_ARG="$2"; shift ;;
+        --variant) VARIANT_ARG="$2"; shift ;;
+        *) echo "Unknown parameter passed: $1"; exit 1 ;;
+    esac
+    shift
+done
+
+# Accept either CLI or env variable overrides
+if [[ -z "$CONFIG" ]]; then
+    CONFIG=${CONFIG_ARG:-"configs/kimi_k3.yaml"}
+fi
+if [[ -z "$VARIANT" ]]; then
+    VARIANT=${VARIANT_ARG:-"all"}
+fi
+
+pip install -qqq hf-transfer
+
+# Run benchmark; use -u to make python prints unbuffered
+python3 -u run_sglang.py --config $CONFIG --model $MODEL --variant $VARIANT
+
+# move the output csv to parent directory
+MODEL_NAME=$(basename $MODEL)
+OUTPUT_CSV="perf_${MODEL_NAME}.csv"
+mv $OUTPUT_CSV ../