[DSv4] Refresh H200/B300/GB300 submissions, densify B300 1k1k sweep / 刷新 H200/B300/GB300 DSv4 基准数据并加密 B300 1k1k 扫描#2189
Conversation
- Bump dsv4-fp8-h200-vllm(+mtp) image v0.21.0 -> v0.25.0 (match #2169) - Bump dsv4-fp8-h200-sglang(+mtp) deepseek-v4-hopper digest to 2026-05-13 push - Densify dsv4-fp4-b300-sglang 1k1k: DP-attn flashinfer band conc 64-512, megamoe band 2048-8192; script selects profiles by CONC range - Re-run entries for dsv4-fp4-b300-trt(+mtp) and dsv4-fp4-gb300-dynamo-trt(+mtp) 中文：刷新 H200/B300/GB300 的 DeepSeek-V4-Pro 配置：H200 vLLM 镜像升级至 v0.25.0，SGLang hopper 镜像更新至 2026-05-13 摘要；加密 B300 SGLang 1k1k 扫描点（DP-attn 64-512、megamoe 2048-8192）；追加 B300 TRT 与 GB300 dynamo-trt 重跑条目。
Oseltamivir
requested review from
Ankur-singh,
jgangani and
kedarpotdar-nv
as code owners
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
2 similar comments
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
I didn't find any bugs, but configs/nvidia-master.yaml is owned by CODEOWNERS (ankur-singh, kedarpotdar-nv, jgangani), so this should get their sign-off per the repo's PR checklist before merging.
What was reviewed: the CONC-range refactor in dsv4_fp4_b300_sglang.sh (exact-match to -le/-ge range bounds) lines up with the new conc-start/conc-end bands added in nvidia-master.yaml, so the script and config changes are internally consistent. The megamoe 2048/4096 launch profiles were already present and are just being reused for the extended search-space, not modified. The two candidate concerns raised by the bug-hunting system (reusing 8k1k-tuned SWA/mem profiles for the new 1k1k conc bands, and the vLLM v0.25.0 image bump reusing gpu-memory-utilization=0.95) were both examined and ruled out as non-issues.
Extended reasoning...
This PR is a routine DSv4 benchmark refresh: bumping pinned images (vLLM v0.21.0→v0.25.0, an SGLang digest), re-running stale/GB300 entries with no config change, and densifying the B300 1k1k concurrency sweep by widening two search-space bands to reuse existing launch profiles. The one behavioral code change is in dsv4_fp4_b300_sglang.sh, converting exact-match CONC checks to range checks so the new densified conc values (64,128,256 and 2048,4096) route to the correct existing profile block — no new profile logic was added.
Security risks
None — this only touches benchmark sweep configuration (image tags/digests, concurrency search-space ranges) and a shell script that selects launch flags for a benchmark server. No auth, secrets, or user-facing code paths are involved.
Level of scrutiny
Low-to-moderate is appropriate: it's benchmark data, not production logic, and the sweep infra (full-sweep-fail-fast label, validate_perf_changelog.py) will catch structural errors before merge. However configs/nvidia-master.yaml is CODEOWNER-owned, so per the repo's own PR checklist a CODEOWNER sign-off is expected regardless of how mechanical the diff looks.
Other factors
The bug-hunting system found no bugs, and I independently checked that the script's new range conditions and the config's new conc-start/conc-end bands agree, and that the reused megamoe profiles are unmodified. No outstanding review comments need addressing.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29310409436 |
1 participant
Summary
Part of the DSv4 (DeepSeek-V4-Pro) submission refresh pass — stale/noisy SKUs only (the GB200 1k1k restore and the missing H100/MI300X/MI325X FP8 recipes will follow in separate PRs).
dsv4-fp8-h200-vllm(+mtp): bump vLLM imagev0.21.0→v0.25.0, matching the B200/B300 dsv4 vLLM bump ([chore] Bump dsv4-fp4 vLLM image to v0.25.0 on B200 and B300 #2169). No script changes needed — the H200 recipe is the FP8/Hopper path and doesn't use the fp4/mega_moe flags that needed--prefill-schedule-intervalon B200/B300.dsv4-fp8-h200-sglang(+mtp): bump the pinnedlmsysorg/sglang:deepseek-v4-hopperdigest from the 2026-05-02 push (7f19c6dc) to the current 2026-05-13 push (1bf5d508).dsv4-fp4-b300-sglangsampled only conc {1, 32, 512, 8192} at 1k1k, leaving the ~50 tok/s/user frontier region unsampled (site reads 933 tok/s/GPU vs B200's 1,359 at 50 tok/s/user while B300 wins at 30 and 100 — likely a density artifact). Added the DP-attn flashinfer_mxfp4 band conc 64–512 and extended the megamoe band to 2048–8192 (reusing the script's existing 2048/4096 launch profiles).dsv4_fp4_b300_sglang.shnow selects launch profiles by CONC range instead of exact match.dsv4-fp4-b300-trt(+mtp)re-run entries (no config change, last run 2026-06-11) so the density check compares same-pass data.dsv4-fp4-b300-sglang-mtpstays capped at conc ≤32 (B300 EAGLE draft CUDA-graph crash at bs≥128, see KLAUD_DEBUG.md), and the TRT 1k1k conc-2048 point stays dropped (MLA-overlap crash, Sync dsv4-fp4-b300-trt recipes with B300 agg frontier config #1703).dsv4-fp4-gb300-dynamo-trt(+mtp)(no config change) — currently the only GB300 dsv4 1k1k submissions.Validation:
generate_sweep_configs.pyproduces the expected densified 1k1k matrix (conc 1, 32, 64, 128, 256, 512, 2048, 4096, 8192);validate_perf_changelog.py --base-ref mainpasses.中文说明
本 PR 属于 DeepSeek-V4-Pro（DSv4）基准数据刷新批次，仅覆盖数据过期/噪声较大的 SKU（GB200 1k1k 恢复以及 H100/MI300X/MI325X 缺失的 FP8 配方将在后续 PR 中单独提交）。
dsv4-fp8-h200-vllm(+mtp)：vLLM 镜像从v0.21.0升级到v0.25.0，与 B200/B300 的 dsv4 vLLM 升级（[chore] Bump dsv4-fp4 vLLM image to v0.25.0 on B200 and B300 #2169）保持一致。无需修改脚本——H200 配方走 FP8/Hopper 路径，不使用 B200/B300 上需要--prefill-schedule-interval的 fp4/mega_moe 参数。dsv4-fp8-h200-sglang(+mtp)：将固定的lmsysorg/sglang:deepseek-v4-hopper镜像摘要从 2026-05-02 推送（7f19c6dc）更新为当前 2026-05-13 推送（1bf5d508）。dsv4-fp4-b300-sglang在 1k1k 下只采样 conc {1, 32, 512, 8192}，导致约 50 tok/s/user 交互性区域完全未采样（官网显示 933 tok/s/GPU，低于 B200 的 1,359，而 B300 在 30 和 100 处均领先——大概率是采样密度问题）。新增 DP-attn flashinfer_mxfp4 频段 conc 64–512，并将 megamoe 频段扩展为 2048–8192（复用脚本中已有的 2048/4096 启动配置）。dsv4_fp4_b300_sglang.sh改为按 CONC 区间选择启动配置。同时为dsv4-fp4-b300-trt(+mtp)追加重跑条目（配置不变，最后一次运行 2026-06-11），保证密度检查基于同一批次数据。dsv4-fp4-b300-sglang-mtp保持 conc ≤32 上限（B300 EAGLE 草稿模型在 bs≥128 时 CUDA graph 崩溃，见 KLAUD_DEBUG.md）；TRT 1k1k conc-2048 采样点保持移除状态（MLA-overlap 崩溃，Sync dsv4-fp4-b300-trt recipes with B300 agg frontier config #1703）。dsv4-fp4-gb300-dynamo-trt(+mtp)追加重跑条目（配置不变）——这是目前 GB300 dsv4 仅有的 1k1k 提交来源。验证：
generate_sweep_configs.py生成的 1k1k 矩阵符合预期（conc 1, 32, 64, 128, 256, 512, 2048, 4096, 8192）；validate_perf_changelog.py --base-ref main通过。