ROCm · raviguptaamd · Aug 5, 2026 · Aug 5, 2026 · Aug 5, 2026 · Aug 5, 2026
@@ -0,0 +1,72 @@
+# Kimi-K3 (MXFP4) serving on AMD Instinct MI300X (gfx942)
+
+[Kimi-K3](https://huggingface.co/moonshotai/Kimi-K3) is Moonshot AI's ~2.8T-parameter
+Mixture-of-Experts model (natively MXFP4/QAT, hybrid MLA + Kimi-Delta-Attention, 896
+experts). These are **colocated** (single-instance) vLLM serving recipes for **MI300X /
+gfx942**, complementing MAD's existing single-node gfx950/MI355X K3 recipe.
+
+Why MI300X needs its own recipe: MI300X has 192 GB/GPU, so the ~1.5 TB checkpoint does
+**not** fit a single 8-GPU node under TP8. All recipes here shard across **2 nodes (16×
+MI300X)** using PP2×TP8 (~102 GB/GPU per node). gfx942 also requires
+`VLLM_ROCM_USE_AITER_MLA=0` (the AITER MLA kernel is gfx950-only).
+
+## Image & model
+
+- **Image:** `amdsiloai/vllm:kimi-k3-mi325x-release-v2` — a gfx942 K3 vLLM build. Public
+  and anonymously pullable (`docker pull amdsiloai/vllm:kimi-k3-mi325x-release-v2`); built
+  for MI325X, also runs MI300X. Override with `-e IMAGE=` if you have a newer tag.
+- **Model:** `moonshotai/Kimi-K3` (MXFP4). Place on local NVMe for fast load; pass via `MODEL_DIR`.
+
+## The recipes
+
+| Folder | Parallelism | Expert all2all | MoE path | Use when |
+|--------|-------------|----------------|----------|----------|
+| [`pp2xtp8/`](pp2xtp8/) | PP2×TP8, no EP | — | a16w4 | Simplest baseline; lowest single-user latency. |
+| [`wideep_int4_allgather/`](wideep_int4_allgather/) | PP2×TP8, EP8/node | `allgather_reducescatter` (generic) | a8w4 (`AITER_SITUV2_A8W4=1`) | Expert-parallel without MoRI kernels. |
+| [`wideep_int4_moriep/`](wideep_int4_moriep/) | PP2×TP8, EP8/node | `mori_low_latency` (**MoRI-EP**) | a8w4 (`AITER_SITUV2_A8W4=1`) | MoRI-EP all2all expert dispatch (intra-node EP group). |
+| [`wideep_disagg_2p2d/`](wideep_disagg_2p2d/) | 2P/2D disagg, TP2×DP8 per pool → EP16, no PP | `mori_low_latency` (**MoRI-EP**) + **MoRIIO** KV/state transfer | MXFP4 | Prefill/decode disaggregation across 4 nodes. Highest concurrent throughput; NIAH validated to 300K. |
+
+The first three are colocated (single-instance, no prefill/decode split); the
+fourth splits prefill and decode across two pools.
+
+**`wideep_disagg_2p2d/` is validated** — single-needle NIAH passes deterministically
+through **300K tokens** (all depths). Two root-cause fixes closed the decode-recall
+bug (4-KV-group block routing + multi-chunk prefill transfer); see its
+[`STATUS.md`](wideep_disagg_2p2d/STATUS.md) and
+[`RESULTS.md`](wideep_disagg_2p2d/RESULTS.md). **Pick by workload:** the colocated
+recipes give the lowest single-request latency (one request spans all 16 GPUs) —
+best for interactive / low-QPS; disagg gives **5.7× throughput at concurrency 8**
+(7.3× at 16) plus decode-latency isolation, at ~4× higher single-stream latency —
+best for batch / high-QPS.
+
+> **EP scope:** the 896 experts split **8-way across each node's 8 GPUs** (112 experts/GPU → `[EP Rank x/8]`), and that EP8 group is replicated on each of the 2 pipeline stages. The expert all2all (incl. MoRI-EP) therefore runs **intra-node**; the only cross-node traffic is the PP activation hand-off, over NCCL. ("16" is the GPU count, not the EP width.)
+
+## Quick start
+
+Each recipe is self-contained (`run.sh` + `README.md` + `niah_probe.py`). Launch the
+**worker (rank 1) first, then the head (rank 0)**:
+
+```bash
+cd <recipe>/
+MODEL_DIR=/path/to/Kimi-K3-MXFP4 ROLE=worker MASTER=<head eth0 IP> bash run.sh   # node 1
+MODEL_DIR=/path/to/Kimi-K3-MXFP4 ROLE=head   MASTER=<head eth0 IP> bash run.sh   # node 0 (API :8000)
+
+# verify
+curl http://<head eth0 IP>:8000/v1/models
+python3 niah_probe.py --url http://<head eth0 IP>:8000 --model kimi-k3 --ctx 8500 --depths 0.1,0.5,0.9
+```
+
+See each recipe's `README.md` for its specific flags and env.
+
+## Common env (set in every run.sh)
+
+- `VLLM_ROCM_USE_AITER_MLA=0` — **required** on gfx942.
+- `--trust-remote-code --reasoning-parser kimi_k3 --mm-encoder-tp-mode data
+  --safetensors-load-strategy prefetch`.
+- ROCm 7.2.x: `HSA_ENABLE_IPC_MODE_LEGACY=0`, `PYTORCH_ALLOC_CONF` /
+  `PYTORCH_HIP_ALLOC_CONF=expandable_segments:False`.
+- **RDMA fabric env is cluster-specific** (`NCCL_IB_HCA`, `NCCL_IB_GID_INDEX`, and the
+  `MORI_*` knobs in the moriep recipe) — adjust for your cluster's NICs/GIDs.
+
+Overridable per recipe: `IMAGE`, `MODEL_DIR`, `MASTER`, `PORT` (8000), `MAX_MODEL_LEN`
+(10240), `MAX_NUM_SEQS` (8), `GPU_UTIL` (0.90).
@@ -0,0 +1,45 @@
+# Kimi-K3 (MXFP4) on MI300X / gfx942 — PP2×TP8 baseline
+
+Serves Kimi-K3 (MXFP4) across **2 nodes** (16× MI300X): tensor-parallel 8 within each
+node, pipeline-parallel 2 across nodes. Each node holds half the layers (~102 GB/GPU);
+a single 8-GPU node cannot fit the model + KV. This is the simplest, lowest-latency K3
+serve on MI300X, with no expert parallelism. See `../wideep_int4_moriep` for wide-EP.
+
+- Image: `amdsiloai/vllm:kimi-k3-mi325x-release-v2` (gfx942 K3 vLLM build; public, anonymous pull)
+- Model: `moonshotai/Kimi-K3` (MXFP4), on local NVMe or NFS
+- Colocated (single instance; no prefill/decode disaggregation)
+
+## Run (worker FIRST, then head)
+
+```bash
+# on the worker node (rank 1):
+MODEL_DIR=/path/to/Kimi-K3-MXFP4 ROLE=worker MASTER=<head eth0 IP> bash run.sh
+
+# on the head node (rank 0, serves the API on :8000):
+MODEL_DIR=/path/to/Kimi-K3-MXFP4 ROLE=head   MASTER=<head eth0 IP> bash run.sh
+```
+
+- `MASTER` = the head node's **eth0** IP (NCCL/PP bootstrap).
+- `MODEL_DIR` = path to the Kimi-K3-MXFP4 weights. Prefer **local NVMe** over NFS (much faster load).
+- First start recompiles gfx942 AITER kernels (a few minutes), then cached.
+
+## Verify
+
+```bash
+curl http://<head eth0 IP>:8000/v1/models
+python3 niah_probe.py --url http://<head eth0 IP>:8000 --model kimi-k3 --ctx 8500 --depths 0.1,0.5,0.9
+```
+
+## Key env (set in run.sh)
+
+| Var | Value | Why |
+|-----|-------|-----|
+| `VLLM_ROCM_USE_AITER_MLA` | `0` | Required on gfx942 — the AITER MLA kernel is gfx950-only and asserts at TP8. |
+| `NCCL_SOCKET_IFNAME` / `GLOO_SOCKET_IFNAME` | `eth0` | Control-plane NIC. |
+| `NCCL_IB_HCA` | `mlx5_0,2,3,4,5,7,8,9` | 8× RDMA NICs (**cluster-specific — override for your fabric**). |
+| `NCCL_IB_GID_INDEX` | `3` | RoCE GID (**cluster-specific**). |
+| `HSA_ENABLE_IPC_MODE_LEGACY` | `0` | ROCm 7.2.x IPC. |
+| `PYTORCH_(HIP_)ALLOC_CONF` | `expandable_segments:False` | Required on ROCm 7.2.x. |
+
+Overridable env: `IMAGE`, `MODEL_DIR`, `MASTER`, `PORT` (8000), `MAX_MODEL_LEN` (10240),
+`MAX_NUM_SEQS` (8), `GPU_UTIL` (0.90).
@@ -0,0 +1,73 @@
+#!/usr/bin/env python3
+"""Lightweight NIAH (needle-in-a-haystack) + basic accuracy probe for a served
+Kimi-K3 endpoint. No deps beyond stdlib (urllib). Run from any node that can
+reach the head IP:port.
+
+Usage:
+  python3 niah_probe.py --url http://<head-ip>:8000 --model kimi-k3 \
+      --ctx 8000 --depths 0.1,0.5,0.9
+"""
+import argparse, json, time, urllib.request, urllib.error
+
+FILLER = ("The quick brown fox jumps over the lazy dog. "
+          "Grand vistas of rolling hills stretched to the horizon. ")
+
+def chat(url, model, prompt, max_tokens=256, timeout=180):
+    body = json.dumps({
+        "model": model,
+        "messages": [{"role": "user", "content": prompt}],
+        "max_tokens": max_tokens, "temperature": 0,
+    }).encode()
+    req = urllib.request.Request(url.rstrip("/") + "/v1/chat/completions",
+                                 data=body, headers={"Content-Type": "application/json"})
+    t0 = time.time()
+    with urllib.request.urlopen(req, timeout=timeout) as r:
+        d = json.load(r)
+    dt = time.time() - t0
+    ch = d["choices"][0]["message"]
+    txt = (ch.get("content") or "") + " " + (ch.get("reasoning") or "")
+    return txt.strip(), d.get("usage", {}), dt
+
+def make_haystack(approx_words, needle, depth):
+    # Build filler to approx word count; insert needle at fractional depth.
+    words = []
+    while len(words) < approx_words:
+        words.extend(FILLER.split())
+    n = len(words)
+    pos = int(n * depth)
+    needle_words = needle.split()
+    words[pos:pos] = needle_words
+    return " ".join(words)
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--url", required=True)
+    ap.add_argument("--model", default="kimi-k3")
+    ap.add_argument("--ctx", type=int, default=8000, help="approx haystack words")
+    ap.add_argument("--depths", default="0.1,0.5,0.9")
+    args = ap.parse_args()
+
+    print(f"== sanity ==")
+    txt, usage, dt = chat(args.url, args.model, "Reply with only: OK", max_tokens=8)
+    print(f"  resp={txt!r} usage={usage} {dt:.1f}s")
+
+    secret = "The secret passcode is HELIOTROPE-7492."
+    question = ("\n\nQuestion: What is the secret passcode? "
+                "Answer with only the passcode.")
+    passed = 0; total = 0
+    for depth in [float(x) for x in args.depths.split(",")]:
+        hay = make_haystack(args.ctx, secret, depth)
+        prompt = hay + question
+        try:
+            txt, usage, dt = chat(args.url, args.model, prompt, max_tokens=256)
+        except Exception as e:
+            print(f"  depth={depth}: ERROR {e}")
+            total += 1; continue
+        ok = "HELIOTROPE-7492" in txt.upper()
+        passed += ok; total += 1
+        print(f"  depth={depth:>4}: {'PASS' if ok else 'FAIL'} "
+              f"(prompt_tok={usage.get('prompt_tokens')}, {dt:.1f}s) resp={txt[:60]!r}")
+    print(f"== NIAH {passed}/{total} passed ==")
+
+if __name__ == "__main__":
+    main()
@@ -0,0 +1,63 @@
+#!/bin/bash
+# Kimi-K3 (MXFP4) on MI300X / gfx942 -- PP2xTP8 baseline (no expert parallelism).
+# TP8 within each node, PP2 across 2 nodes -> each node holds half the layers
+# (~102 GB/GPU); a single 8-GPU node cannot fit the model + KV. Simplest, lowest-
+# latency K3 serve on MI300X. Colocated (single instance; no P/D disaggregation).
+#
+# Usage (worker FIRST, then head):
+#   ROLE=worker MASTER=<head eth0 IP> bash run.sh   # on node1 (rank1)
+#   ROLE=head   MASTER=<head eth0 IP> bash run.sh   # on node0 (rank0, serves API)
+set -euo pipefail
+
+IMAGE="${IMAGE:-amdsiloai/vllm:kimi-k3-mi325x-release-v2}"
+MODEL_DIR="${MODEL_DIR:?set MODEL_DIR=<path to Kimi-K3-MXFP4 weights>}"
+ROLE="${ROLE:?set ROLE=head|worker}"
+MASTER="${MASTER:?set MASTER=<head eth0 IP>}"
+PORT="${PORT:-8000}"
+MAX_MODEL_LEN="${MAX_MODEL_LEN:-10240}"
+MAX_NUM_SEQS="${MAX_NUM_SEQS:-8}"
+GPU_UTIL="${GPU_UTIL:-0.90}"
+CONTAINER="k3_pp2tp8_${ROLE}"
+LOGHOST="${LOGHOST:-$HOME/k3run/logs}"; mkdir -p "$LOGHOST"
+
+# Proven fabric env (from cluster_rdma_env_recommender.py)
+BOOT_NIC="eth0"
+IB_HCA="mlx5_0,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_7,mlx5_8,mlx5_9"
+GID_INDEX=3
+
+if [ "$ROLE" = "head" ]; then
+  NODE_RANK=0; SERVE_EXTRA="--port ${PORT}"; HEADLESS=""
+else
+  NODE_RANK=1; SERVE_EXTRA=""; HEADLESS="--headless"
+fi
+
+echo "[pp2tp8] node=$(hostname -s) role=$ROLE rank=$NODE_RANK master=$MASTER"
+[ -f "$MODEL_DIR/model.safetensors.index.json" ] || { echo "ERROR: model missing at $MODEL_DIR"; exit 1; }
+docker rm -f "$CONTAINER" 2>/dev/null || true
+
+docker run -d --name "$CONTAINER" \
+  --network host --ipc host \
+  --device /dev/kfd --device /dev/dri --device /dev/infiniband --group-add video \
+  --cap-add SYS_PTRACE --security-opt seccomp=unconfined --shm-size 128g \
+  --ulimit memlock=-1 --ulimit stack=67108864 \
+  -e VLLM_ROCM_USE_AITER_MLA=0 \
+  -e NCCL_SOCKET_IFNAME=$BOOT_NIC -e GLOO_SOCKET_IFNAME=$BOOT_NIC \
+  -e NCCL_IB_DISABLE=0 -e NCCL_IB_HCA=$IB_HCA -e NCCL_IB_GID_INDEX=$GID_INDEX \
+  -e NCCL_IGNORE_CPU_AFFINITY=1 -e NCCL_DEBUG=WARN \
+  -e HSA_ENABLE_IPC_MODE_LEGACY=0 -e HSA_NO_SCRATCH_RECLAIM=1 \
+  -e PYTORCH_ALLOC_CONF=expandable_segments:False \
+  -e PYTORCH_HIP_ALLOC_CONF=expandable_segments:False \
+  -v "$MODEL_DIR":/model:ro -v "$LOGHOST":/logs \
+  --entrypoint bash \
+  "$IMAGE" -c "
+    vllm serve /model --served-model-name kimi-k3 \
+      --tensor-parallel-size 8 --pipeline-parallel-size 2 \
+      --distributed-executor-backend mp \
+      --nnodes 2 --node-rank ${NODE_RANK} --master-addr ${MASTER} --master-port 29500 ${HEADLESS} \
+      --trust-remote-code --reasoning-parser kimi_k3 --mm-encoder-tp-mode data \
+      --safetensors-load-strategy prefetch \
+      --max-model-len ${MAX_MODEL_LEN} --max-num-seqs ${MAX_NUM_SEQS} \
+      --gpu-memory-utilization ${GPU_UTIL} ${SERVE_EXTRA} 2>&1 | tee /logs/vllm_pp2tp8_${ROLE}.log
+  "
+echo "[pp2tp8] $ROLE started. log: $LOGHOST/vllm_pp2tp8_${ROLE}.log"
+[ "$ROLE" = head ] && echo "[pp2tp8] health: curl http://${MASTER}:${PORT}/v1/models"