Add Kimi K2.6 NVFP4 B300 EAGLE3 AgentX benchmark / 新增 Kimi K2.6 NVFP4 B300 EAGLE3 AgentX 基准测试

[xinli-sw]

No description provided.

[github-actions]

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-fail-fast label (strongly recommended) to this PR — the benchmark sweep only runs on labeled PRs. Use full-sweep-enabled only if you need matrix jobs to keep running past a failure.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs

---

感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have。

如需进行 PR 验证，请为此 PR 添加 full-sweep-fail-fast 标签（强烈推荐）— 基准测试 sweep 仅在带有标签的 PR 上运行。仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled。

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档

[claude]

🔴 The new kimik2.5-fp4-b300-vllm-agentic-mtp config sets runner: cluster:b300 at configs/nvidia-master.yaml:2985, but that label doesn't exist in configs/runners.yaml — only cluster:b300-nv and cluster:b300-cw are defined. Because one search-space entry uses kv-offloading: dram, matrix generation calls agentic_dram_offload_gb → runner_available_cpu_dram_mib('cluster:b300', ...) which raises ValueError (no hardware entry), aborting sweep-config generation before any benchmark runs. Change to cluster:b300-nv to match the sibling non-mtp config at line 2970.

Extended reasoning...

What the bug is

The new kimik2.5-fp4-b300-vllm-agentic-mtp recipe added in this PR (configs/nvidia-master.yaml:2981-2996) declares runner: cluster:b300 at line 2985. This label does not exist anywhere in configs/runners.yaml. Grepping the runners file shows only two B300 cluster labels are defined: cluster:b300-cw (line 216) and cluster:b300-nv (line 222), and only cluster:b300-nv has a hardware: entry (line 306). The sibling non-mtp recipe kimik2.5-fp4-b300-vllm-agentic at line 2970 correctly uses cluster:b300-nv on what is clearly the same physical fleet.

How it manifests

The recipe contains a search-space entry with kv-offloading: dram, kv-offload-backend: native (line 2995). In utils/matrix_logic/generate_sweep_configs.py, generate_full_sweep calls agentic_dram_offload_gb(agentic_config, bmk, runner, runner_data) for that entry. That function calls runner_available_cpu_dram_mib('cluster:b300', runner_data) → runner_hardware_int(...) which looks up hardware['cluster:b300']. Since there is no such key (only cluster:b300-nv is registered under hardware:), runner_hardware_int raises:

ValueError: Runner 'cluster:b300' requires 'available-cpu-dram-mib' in runner hardware metadata.
Available hardware keys: cluster:b300-nv, ...

That exception propagates out of generate_full_sweep, so the entire sweep-config generation step (the get-jobs/setup job in the workflow) fails with a non-zero exit code. No benchmark job in the matrix ever dispatches — not even the non-dram entries at lines 2993/2994/2996.

Why existing code doesn't catch it

AgenticCodingConfig.validate_dram_offload_capacity in validation.py only checks that dram-utilization is set when kv-offloading == 'dram'; it does not cross-check that the recipe's runner: label actually appears in runners.yaml. Master-config validation is purely structural, so an arbitrary string is accepted for runner. The mistake only surfaces at matrix-generation time.

Impact

This is the whole point of the PR: the sweep for the new MTP recipe cannot generate configs, so no runs will occur. Even for the non-dram entries where the missing-hardware code path is not exercised, GitHub Actions would try to schedule jobs with runs-on: cluster:b300, which no self-hosted runner group carries — so they would sit in "waiting for a runner" forever and eventually time out.

Fix

One-character change: replace runner: cluster:b300 with runner: cluster:b300-nv at configs/nvidia-master.yaml:2985, matching the sibling recipe at line 2970. If the CW fleet was intended instead, use cluster:b300-cw — but note that cluster:b300-cw has no hardware: entry either, so the dram search-space entry would still fail; only cluster:b300-nv currently supports DRAM offload matrix generation.

Step-by-step proof

1. configs/runners.yaml line 306: hardware: only contains cluster:b300-nv (no cluster:b300).
2. run-sweep.yml / e2e-tests.yml 'setup' job runs python3 utils/matrix_logic/generate_sweep_configs.py ....
3. generate_full_sweep iterates configs; reaches kimik2.5-fp4-b300-vllm-agentic-mtp, is_multinode = False, runner = "cluster:b300".
4. Enters the agentic loop; for the dram entry {tp: 4, ep: 1, spec-decoding: mtp, kv-offloading: dram, kv-offload-backend: native, conc-list: [2, 4, 16]}.
5. total_cpu_dram_gb = agentic_dram_offload_gb(agentic_config, bmk, "cluster:b300", runner_data).
6. agentic_dram_offload_gb → runner_available_cpu_dram_mib("cluster:b300", runner_data) → runner_hardware_int("cluster:b300", runner_data, "available-cpu-dram-mib").
7. runner_hardware(runner_data).get("cluster:b300", {}) returns {}; value = None → raises ValueError.
8. Python exits non-zero; CONFIG_JSON=$(python3 ...) fails; the get-jobs/setup step fails; no downstream benchmark job dispatches.

[functionstackx]

thanks for the contribution @xinli-sw

can u please update recipes.vllm.ai with ur new configs?

vllm recipes doesn't currently support offloading tho the inferact guys said they are open to contributions to support mooncake store, lmcache, vllm native offloading support in vllm recipes

as in sglang cookbook recipes , they also have support for cpu offloading recipes

similar applies to DCP, doubt the inferact folks won't mind contributions to add DCP support too

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29121027372
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29121027372

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29137748102
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29137748102

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29138755696
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29138755696

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29139024185
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29139024185

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29140751554
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29140751554

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29157674951
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29157674951

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29158176591
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29158176591

[Ankur-singh]

Opened this PR for the recipe. Will work with the vLLM team to get it in. Feel free to comment and review the PR, and help me get the right recipe in.

vllm-project/recipes#626

[Ankur-singh]

/reuse-sweep-run

[Ankur-singh]

As a PR reviewer and CODEOWNER, I have reviewed this and have:

• Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
• Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
• Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this.
• Verified that this PR passes evals. Please link to GitHub Action workflow that shows this.
• Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
• For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
• Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; FLOPs at lower precisions is fine, given that the config passes private evals. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
• If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
• If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
• Verified that the single-node recipes are similar to the official vLLM recipes and/or theSGLang cookbook:
  • If they are not, I have verified that a PR has been opened in vLLM recipe repo or SGLang repo and linked it below in the additional detail section:
• Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/<PR_NUMBER>.md — named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section.
• If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.

Additional detail section:

• PR validation: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/29158176591 / PR 验证：https://github.com/SemiAnalysisAI/InferenceX/actions/runs/29158176591
• Evals: not applicable to this AgentX-only submission; the eval jobs are intentionally skipped. / 评估：本提交仅包含 AgentX 基准测试，因此不适用；评估任务按设计跳过。
• Upstream vLLM recipe PR: Update Kimi-K2.6 B300 NVFP4 recipe / 更新 Kimi-K2.6 B300 NVFP4 配置 vllm-project/recipes#626 / 上游 vLLM recipe PR：Update Kimi-K2.6 B300 NVFP4 recipe / 更新 Kimi-K2.6 B300 NVFP4 配置 vllm-project/recipes#626
• AgentX uses /v1/chat/completions, satisfying the MTP chat-template requirement. / AgentX 使用 /v1/chat/completions，满足 MTP 的 chat template 要求。
• The non-DCP EAGLE3 path uses draft length 4 with synthetic acceptance length 3.24, matching golden_al_distribution/kimik2.5_eagle3.yaml. / 非 DCP 的 EAGLE3 路径使用 draft length 4 和 synthetic acceptance length 3.24，与 golden_al_distribution/kimik2.5_eagle3.yaml 一致。
• The PR has no description and has an active CHANGES_REQUESTED review. / 该 PR 没有描述，并且仍有有效的 CHANGES_REQUESTED 审阅状态。

Signed: ankur-singh

[functionstackx]

Opened this PR for the recipe. Will work with the vLLM team to get it in. Feel free to comment and review the PR, and help me get the right recipe in.

vllm-project/recipes#626

thanks @Ankur-singh you prob need to add support for CPU KVCache Offloading to vllm recipes too. the LMCache folks have an proposed PR on how to do it that the inferact vllm maintainers r looking vllm-project/recipes#625 maybe u should stack ur recipe PR to extend the LMCache recipe PR to work with vLLM SimpleCPUOffloadConnector ?

VLLM_USE_SIMPLE_KV_OFFLOAD=1
        CPU_OFFLOAD_BYTES=$((TOTAL_CPU_DRAM_GB * 1024 * 1024 * 1024))
        OFFLOAD_ARGS=(
            --disable-hybrid-kv-cache-manager
            --kv-transfer-config
            "{\"kv_connector\":\"SimpleCPUOffloadConnector\",\"kv_role\":\"kv_both\",\"kv_connector_extra_config\":{\"cpu_bytes_to_use\":$CPU_OFFLOAD_BYTES,\"lazy_offload\":false}}"

[Klaud-Cold]

❌❌❌ REJECTED ❌❌❌

@Ankur-singh — blocking: Kimi K2.6 has no committed golden AL curve, and the pinned synthetic_acceptance_length: 3.24 is taken from the Kimi K2.5 curve. golden_al_distribution/kimik2.5_eagle3.yaml is explicitly "Measured on kimi-k2.5-nvfp4" with the k2.5 draft head, while this PR serves Kimi-K2.6-NVFP4 with the different lightseekorg/kimi-k2.6-eagle3-mla draft. Per golden_al_distribution/README.md, different models keep their own SPEED-Bench-derived curves and a submission may not substitute a different acceptance target; the sign-off states no K2.6 AL source (e.g. a speedbench-al.yml collection run). Fix: collect and commit the K2.6 golden curve (or document the pending collection run) and post a fresh sign-off.

• ✅ Check 0 (CODEOWNER): PASS — @ankur-singh owns configs/nvidia-master.yaml; every other changed path carries only the catch-all.
• ✅ Check 1 (sweep on in-PR commit): PASS — run 29158176591 (conclusion success) on head 61d59b7: all 14 agentic / benchmark jobs succeeded and produced bmk_agentic_* artifacts, which validate_reusable_run accepts as a reusable source; the single-node */ and eval / lanes are empty-by-design for this agentic-only config.
• ➖ Check 2 (evals pass): N/A — agentic-coding sweeps run no evals by design (run-eval: false hard-coded in run-sweep.yml); accuracy fairness for agentic spec-decode is instead the golden-AL mechanism, which is what fails in Check 10.
• ✅ Check 3 (recipe linked and complete): PASS — vllm-project/recipes#626 is linked and its serve command matches every major arg of this PR (model, TP4, EAGLE3 draft + num_speculative_tokens: 4, TOKENSPEED_MLA, TRTLLM_RAGGED attention-config, fp8 KV cache, CPU-offload connector, DCP4-without-spec caveat). Concurrency sweeps/ports differ as expected harness tuning.
• ✅ Check 4 (reuse command): PASS — /reuse-sweep-run posted by Ankur-singh (COLLABORATOR).
• ✅ Check 5 (latest template): PASS — all current-template items present; the one unchecked evals item is explained in the additional detail section, as the template's final item permits.
• ✅ Check 6 (upstream image / engine-first): PASS — vllm/vllm-openai:nightly-09663abde… is the upstream vLLM org; framework is vLLM itself, so engine-first ordering is trivially satisfied.
• ✅ Check 7 (no architecture hacks): PASS — no --hf-overrides or architecture knobs; --language-model-only only skips loading the vision encoder that text-only AgentX traffic never exercises (a documented opt-in in the official recipe), and the quantization flags are precision, not architecture.
• ✅ Check 8 (chat template): PASS — the replay client drives /v1/chat/completions with --endpoint-type chat (benchmarks/benchmark_lib.sh).
• ✅ Check 9 (no engine patches): PASS — no patching of the serving stack; aiperf installs into an isolated venv specifically to avoid the server's site-packages.
• ❌ Check 10 (golden AL): FAIL — no committed golden curve exists for Kimi K2.6; the pinned 3.24 is the K2.5 thinking_on / draft-length-4 value, and the sign-off cites the K2.5 file instead of a K2.6 source. (Simulated synthetic acceptance itself is correctly configured on the non-DCP arms; the TP4/DCP4 arms run with speculation disabled entirely pending Add DCP + Eagle support for Tokenspeed MLA and Flashinfer backends vllm-project/vllm#48180.)

Additional notes (non-blocking): the perf-changelog.yaml entry claims the DCP4 points use num_speculative_tokens=3 / AL 2.88, but the script disables EAGLE3 whenever DCP > 1 — please correct the changelog. An active CHANGES_REQUESTED review from functionstackx is also still open on this PR.