Add Kimi-K3 for MI355X: vLLM, SGLang, and ATom

README.md

@@ -14,7 +14,7 @@ Below are blueprints of supported models along with their documentation.
 | [**xDiT diffusion inference**](benchmark/xdit/README.md) | Diffusion Transformer inference using xDiT | FLUX.1, FLUX.1 Kontext, FLUX.2, FLUX.2 Klein, HunyuanVideo, HunyuanVideo 1.5, LTX-2, Stable Diffusion 3.5, Wan 2.1, Wan 2.2, Z-Image Turbo |
 | [**JAX MaxText training**](benchmark/jax_maxtext/README.md) | Train LLMs on AMD Instinct GPUs using JAX MaxText | Llama 2 7B/70B, Llama 3/3.1 8B/70B, Llama 3.1 405B, Llama 3.3 70B, DeepSeek-V2-lite 16B, Mixtral-8x7B |
 | [**vLLM inference**](benchmark/vllm/README.md) | LLM Inference with vLLM on AMD Instinct GPUs | DeepSeek-R1, gpt-oss-20b/120b, Llama-2-70b, Llama-3.1-8b/405b, Llama-3.3-70b, Llama-4-Scout/Maverick, Mixtral-8x7b/8x22b, Phi-4, Qwen3-8b/32b/30b-a3b/235b-a22b |
-| [**SGLang inference**](benchmark/sglang/README.md) | LLM Inference with SGLang on AMD Instinct GPUs | DeepSeek-R1-Distill-Qwen-32B |
+| [**SGLang inference**](benchmark/sglang/README.md) | LLM Inference with SGLang on AMD Instinct GPUs | DeepSeek-R1-Distill-Qwen-32B, Kimi-K3 |
 | [**PyTorch training**](benchmark/pytorch_train/README.md) | Train LLMs on AMD Instinct GPUs using AMD's Primus | Llama 2/3/3.1/3.2/3.3/4, GPT-OSS 20B/120B, Qwen2/2.5/3, Flux, SDXL, DLRM, and others |
 | [**PyTorch inference**](benchmark/pytorch_inference/README.md) | Inference recipes for Multimodal, video and vision transformer models | Mochi video, Chai-1, CLIP (ViT-B-32), Wan2.1, Janus-Pro-7B, HunyuanVideo |
 | [**Megatron-LM training**](benchmark/megatron_lm/README.md) | Train LLMs on AMD Instinct GPUs using ROCm Megatron-LM | Llama 2 7B/70B, Llama 3/3.1 8B/70B, Llama 3.3 70B, DeepSeek-V2-lite, DeepSeek-V3, Mixtral 8x7B/8x22B, Qwen 2.5 7B/72B |

benchmark/sglang/README.md

@@ -80,6 +80,48 @@ users can also change the benchmarking parameters. Refer to the [Standalone benc
 | model_name                              |
 | --------------------------------------- |
 | pyt_sglang_deepseek-r1-distill-qwen-32b  |
+| pyt_sglang_kimi-k3                       |
+| pyt_sglang_kimi-k3_dspark                |
+
+>[!NOTE]
+>The two `pyt_sglang_kimi-k3*` entries are the exception to everything described above. They track the
+>AMD day-0 recipes in [sgl-project/sglang#32548](https://github.com/sgl-project/sglang/issues/32548)
+>(day-0 support: [#32541](https://github.com/sgl-project/sglang/pull/32541), see also the
+>[SGLang K3 cookbook](https://docs.sglang.io/cookbook/autoregressive/Moonshotai/Kimi-K3)) and differ in
+>four ways:
+>
+>- **Image.** They build from `lmsysorg/sglang-rocm:rocm720-mi35x-k3-20260727` via
+>  [docker/pyt_sglang_kimi_k3.ubuntu.amd.Dockerfile](../../docker/pyt_sglang_kimi_k3.ubuntu.amd.Dockerfile),
+>  not the shared `lmsysorg/sglang:v0.4.5-rocm630` above, which predates K3 support.
+>- **Benchmark.** They measure *online serving* (`sglang serve` + `sglang.benchmark.serving`) through
+>  [scripts/sglang/run_sglang.py](../../scripts/sglang/run_sglang.py) and
+>  [scripts/sglang/configs/kimi_k3.yaml](../../scripts/sglang/configs/kimi_k3.yaml), rather than the
+>  offline latency/throughput path documented below.
+>- **Hardware.** 8x MI350X/MI355X (gfx950) TP8 only, hence `skip_gpu_arch: gfx942`. The checkpoint is
+>  large, so make sure `HF_HUB_CACHE` has room; `pyt_sglang_kimi-k3_dspark` additionally pulls the
+>  [RadixArk/Kimi-K3-DSpark](https://huggingface.co/RadixArk/Kimi-K3-DSpark) draft checkpoint.
+>- **Invocation.** They carry no sweep tag, so a tag run does not pull the checkpoint and hold 8 GPUs.
+>  Run them explicitly by name:
+>
+>```sh
+>madengine run --tags pyt_sglang_kimi-k3 --keep-model-dir --live-output
+>madengine run --tags pyt_sglang_kimi-k3_dspark --keep-model-dir --live-output
+>```
+>
+>The sweep is 8192-token input / 1024-token output at concurrency 2/4/8/16/32. The issue does not state
+>its input and output lengths; they were recovered from its own tables, where
+>`(E2EL - TTFT) / TPOT + 1` lands on ~1024 output tokens on every row and
+>`concurrency x (inp + out) / E2EL` reproduces the reported total throughput only at 8192 input tokens.
+>Keeping that shape is what makes MAD's numbers comparable to the issue's.
+>
+>The runner emits `--tp-size` where the issue writes `--tp`. There is no `--tp` server argument;
+>`tp_size` has a single alias, `--tensor-parallel-size`, and `--tp` resolves only through argparse
+>prefix matching, which a future `--tp*` option would silently break.
+
+>[!WARNING]
+>The published performance tables were measured on **MI355X**. A commenter on the tracking issue reports
+>much weaker results on **MI350X** with untuned AITER kernels. Both report as gfx950, so `skip_gpu_arch`
+>cannot distinguish them — treat MI350X numbers from this recipe as unvalidated.
 
 ### Standalone benchmarking              
 -----------------------------

benchmark/vllm/README.md

@@ -104,6 +104,7 @@ users can also directly run the vLLm benchmark scripts and change the benchmarki
 | pyt_vllm_gpt-oss-120b_w4a8             | [amd/gpt-oss120b-w-mxfp4-a-fp8](https://huggingface.co/amd/gpt-oss120b-w-mxfp4-a-fp8) |
 | pyt_vllm_kimi-k2.6                     | [moonshotai/Kimi-K2.6](https://huggingface.co/moonshotai/Kimi-K2.6) |
 | pyt_vllm_kimi-k2.6_fp4                 | [amd/Kimi-K2.6-MXFP4](https://huggingface.co/amd/Kimi-K2.6-MXFP4) |
+| pyt_vllm_kimi-k3                       | [moonshotai/Kimi-K3](https://huggingface.co/moonshotai/Kimi-K3) |
 | pyt_vllm_llama-3.1-8b                  | [meta-llama/Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct) |
 | pyt_vllm_llama-3.1-8b_fp8              | [amd/Llama-3.1-8B-Instruct-FP8-KV](https://huggingface.co/amd/Llama-3.1-8B-Instruct-FP8-KV) |
 | pyt_vllm_llama-3.1-405b                | [meta-llama/Llama-3.1-405B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-405B-Instruct) |
@@ -129,6 +130,25 @@ users can also directly run the vLLm benchmark scripts and change the benchmarki
 | pyt_vllm_qwen3.5-397b-a17b             | [Qwen/Qwen3.5-397B-A17B](https://huggingface.co/Qwen/Qwen3.5-397B-A17B) |
 | pyt_vllm_qwen3.5-397b-a17b_fp8         | [Qwen/Qwen3.5-397B-A17B-FP8](https://huggingface.co/Qwen/Qwen3.5-397B-A17B-FP8) |
 
+>[!NOTE]
+>`pyt_vllm_kimi-k3` is the one exception to the shared Docker image above. Kimi K3 requires
+>vLLM >= 0.27.0, which is not yet in a tagged `vllm-openai-rocm` release, so it builds from
+>the model-specific `vllm/vllm-openai-rocm:kimi-k3` image via
+>[docker/pyt_vllm_kimi_k3.ubuntu.amd.Dockerfile](../../docker/pyt_vllm_kimi_k3.ubuntu.amd.Dockerfile).
+>It needs an 8x MI350X/MI355X (gfx950) node — the ~1680 GB minimum footprint does not fit a
+>single 8x MI300X node — and the checkpoint is ~1.56 TB, so make sure `HF_HUB_CACHE` has room.
+>It is deliberately not tagged `vllm_default`; run it explicitly:
+>
+>```sh
+>madengine run --tags pyt_vllm_kimi-k3 --keep-model-dir --live-output
+>```
+>
+>The config tracks the [MI355X recipe profile](https://recipes.vllm.ai/moonshotai/Kimi-K3?hardware=mi355x)
+>for a text-only serving run, with two intentional deviations: MAD adds
+>`--no-enable-prefix-caching` for benchmark hygiene (as it does for every model here), and the
+>gsm8k accuracy stage is disabled because K3's always-on reasoning is returned inline over
+>`/v1/completions` and exhausts the generation budget.
+
 
 ### Standalone benchmarking              
 -----------------------------

docker/pyt_atom.ubuntu.amd.Dockerfile

@@ -0,0 +1,39 @@
+# CONTEXT {'gpu_vendor': 'AMD', 'guest_os': 'UBUNTU'}
+###############################################################################
+#
+# MIT License
+#
+# Copyright (c) Advanced Micro Devices, Inc.
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all
+# copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+#
+#################################################################################
+ARG BASE_DOCKER=rocm/atom-dev:latest
+FROM $BASE_DOCKER
+
+USER root
+ENV WORKSPACE_DIR=/workspace
+RUN mkdir -p $WORKSPACE_DIR
+WORKDIR $WORKSPACE_DIR
+
+# record configuration for posterity
+RUN pip3 list
+
+# Specify entrypoint to override upstream
+ENTRYPOINT [""]

docker/pyt_sglang_kimi_k3.ubuntu.amd.Dockerfile

@@ -0,0 +1,50 @@
+# CONTEXT {'gpu_vendor': 'AMD', 'guest_os': 'UBUNTU'}
+###############################################################################
+#
+# MIT License
+#
+# Copyright (c) 2025 Advanced Micro Devices, Inc.
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all
+# copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+#
+#################################################################################
+# Kimi K3 day-0 support landed in sgl-project/sglang#32541 and is not in a
+# tagged SGLang ROCm release yet; this model-specific image is the only ROCm
+# build carrying the KDA / Stable LatentMoE / AITER A8W4 support the checkpoint
+# needs. Kept separate from docker/pyt_sglang, which is still on the v0.4.5
+# rocm630 base that the existing SGLang entry is validated against.
+#
+# The tag is the day-0 image named in the AMD tracking issue
+# https://github.com/sgl-project/sglang/issues/32548, which is what its MI355X
+# performance tables were measured against. A newer rocm720-mi35x-k3-20260728
+# tag exists on Docker Hub but nothing published ties it to the recipe, so it is
+# deliberately not adopted here.
+#
+# Fold this back into docker/pyt_sglang once K3 lands in a versioned ROCm image.
+ARG BASE_DOCKER=lmsysorg/sglang-rocm:rocm720-mi35x-k3-20260727
+
+FROM $BASE_DOCKER
+
+USER root
+ENV WORKSPACE_DIR=/workspace
+RUN mkdir -p $WORKSPACE_DIR
+WORKDIR $WORKSPACE_DIR
+
+# record configuration for posterity
+RUN pip3 list

docker/pyt_vllm_kimi_k3.ubuntu.amd.Dockerfile

@@ -0,0 +1,44 @@
+# CONTEXT {'gpu_vendor': 'AMD', 'guest_os': 'UBUNTU'}
+###############################################################################
+#
+# MIT License
+#
+# Copyright (c) Advanced Micro Devices, Inc.
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all
+# copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+#
+#################################################################################
+# Kimi K3 requires vLLM >= 0.27.0, which is not in a tagged vllm-openai-rocm
+# release yet; the model-specific :kimi-k3 image is the only ROCm build with the
+# KDA / Gated MLA / Stable LatentMoE support the checkpoint needs. Kept separate
+# from docker/pyt_vllm so the ~35 other vLLM entries stay on the tagged release.
+# Fold this back into docker/pyt_vllm once K3 lands in a vX.Y.Z ROCm image.
+ARG BASE_DOCKER=vllm/vllm-openai-rocm:kimi-k3
+FROM $BASE_DOCKER
+
+USER root
+ENV WORKSPACE_DIR=/workspace
+RUN mkdir -p $WORKSPACE_DIR
+WORKDIR $WORKSPACE_DIR
+
+# record configuration for posterity
+RUN pip3 list
+
+# Specify entrypoint to override upstream
+ENTRYPOINT [""]

models.json

@@ -327,6 +327,47 @@
     "args":
      "--model_repo amd/Kimi-K2.6-MXFP4 --config configs/default.yaml"
   },
+  {
+    "name": "pyt_vllm_kimi-k3",
+    "url": "",
+    "dockerfile": "docker/pyt_vllm_kimi_k3",
+    "scripts": "scripts/vllm/run.sh",
+    "data": "huggingface",
+    "n_gpus": "-1",
+    "owner": "mad.support@amd.com",
+    "training_precision": "",
+    "multiple_results": "perf_Kimi-K3.csv",
+    "tags": [
+      "pyt",
+      "vllm",
+      "inference"
+    ],
+    "timeout": -1,
+    "skip_gpu_arch": "gfx942",
+    "args":
+     "--model_repo moonshotai/Kimi-K3 --config configs/default.yaml"
+  },
+  {
+    "name": "pyt_atom_kimi-k3",
+    "url": "",
+    "dockerfile": "docker/pyt_atom",
+    "scripts": "scripts/atom/run.sh",
+    "data": "huggingface",
+    "n_gpus": "-1",
+    "owner": "mad.support@amd.com",
+    "training_precision": "",
+    "multiple_results": "perf_Kimi-K3.csv",
+    "tags": [
+      "pyt",
+      "atom",
+      "atom_default",
+      "inference"
+    ],
+    "timeout": -1,
+    "skip_gpu_arch": "gfx942, gfx906, gfx908, gfx90a, A100, H100, V100",
+    "args":
+     "--model_repo moonshotai/Kimi-K3 --config configs/default.yaml"
+  },
   {
     "name": "pyt_vllm_llama-3.1-8b",
     "url": "",
@@ -2194,6 +2235,46 @@
     "args":
      "--model_repo deepseek-ai/DeepSeek-R1-Distill-Qwen-32B --test_option latency --num_gpu 8 --datatype bfloat16 --dataset random --batch_size 1,8,32 --lat_input_output_len '128:128;128:1024;1024:128;1024:1024'"
   },
+  {
+    "name": "pyt_sglang_kimi-k3",
+    "url": "",
+    "dockerfile": "docker/pyt_sglang_kimi_k3",
+    "scripts": "scripts/sglang/run_kimi_k3.sh",
+    "data": "huggingface",
+    "n_gpus": "-1",
+    "owner": "mad.support@amd.com",
+    "training_precision": "",
+    "multiple_results": "perf_Kimi-K3.csv",
+    "tags": [
+      "pyt",
+      "sglang",
+      "inference"
+    ],
+    "timeout": -1,
+    "skip_gpu_arch": "gfx942",
+    "args":
+     "--model_repo moonshotai/Kimi-K3 --config configs/kimi_k3.yaml --variant nospec"
+  },
+  {
+    "name": "pyt_sglang_kimi-k3_dspark",
+    "url": "",
+    "dockerfile": "docker/pyt_sglang_kimi_k3",
+    "scripts": "scripts/sglang/run_kimi_k3.sh",
+    "data": "huggingface",
+    "n_gpus": "-1",
+    "owner": "mad.support@amd.com",
+    "training_precision": "",
+    "multiple_results": "perf_Kimi-K3.csv",
+    "tags": [
+      "pyt",
+      "sglang",
+      "inference"
+    ],
+    "timeout": -1,
+    "skip_gpu_arch": "gfx942",
+    "args":
+     "--model_repo moonshotai/Kimi-K3 --config configs/kimi_k3.yaml --variant dspark"
+  },
   {
     "name": "pyt_hy_video",
     "url": "",

scripts/atom/configs/accuracy.yaml

@@ -0,0 +1,17 @@
+# ATOM Kimi-K3 accuracy benchmark (GSM8K 5-shot)
+# Expected range: flexible-extract 0.9538–0.9591
+
+- benchmark: accuracy
+  model: moonshotai/Kimi-K3
+  tp: 8
+  kv_cache_dtype: fp8
+  num_fewshot: 5
+  apply_chat_template: false
+  extra_args:
+    --trust-remote-code: true
+    --max-model-len: 16384
+    --max-num-seqs: 64
+    --max-num-batched-tokens: 16384
+    --gpu-memory-utilization: 0.93
+    --block-size: 128
+    --no-enable_prefix_caching: true

scripts/atom/configs/default.yaml

@@ -0,0 +1,20 @@
+# ATOM Kimi-K3 serving benchmark configs
+# 1k/4k ISL, 1k OSL, mc64/128/256
+# gfx950 (MI350X/MI355X) only — MXFP4 weights require gfx95x
+
+## Kimi-K3 MXFP4 TP8
+- benchmark: serving
+  model: moonshotai/Kimi-K3
+  tp: 8
+  inp: 1024 4096
+  out: 1024
+  kv_cache_dtype: fp8
+  max_concurrency: 64 128 256
+  extra_args:
+    --trust-remote-code: true
+    --max-model-len: 16384
+    --max-num-seqs: 64
+    --max-num-batched-tokens: 16384
+    --gpu-memory-utilization: 0.93
+    --block-size: 128
+    --no-enable_prefix_caching: true

scripts/atom/configs/perf.yaml

@@ -0,0 +1,25 @@
+# ATOM Performance Recipes
+# Source: https://github.com/ROCm/ATOM/pull/1718 (recipes/Kimi-K3.md)
+# Merge strategy: User config wins; perf fills gaps (missing keys only)
+
+# Kimi-K3 MXFP4 (gfx950 / MI355)
+- model: moonshotai/Kimi-K3
+  extra_args:
+    --kv_cache_dtype: fp8
+    --trust-remote-code: true
+    --max-model-len: 16384
+    --max-num-seqs: 64
+    --max-num-batched-tokens: 16384
+    --gpu-memory-utilization: 0.93
+    --block-size: 128
+    --no-enable_prefix_caching: true
+  env:
+    ATOM_LOADER_USE_THREADPOOL: '1'
+    ATOM_LOADER_THREADPOOL_WORKERS: '16'
+    ATOM_SYNC_AFTER_LOAD: '1'
+    ATOM_DIST_TIMEOUT_SECONDS: '3600'
+    ATOM_USE_TRITON_GEMM: '1'
+    AITER_USE_GROUPED_GEMM: '0'
+    ATOM_USE_TRITON_MOE: '0'
+    AITER_FLYDSL_FORCE: '1'
+    AITER_FORCE_GFX1250: '0'