Skip to content

[DSv4] Refresh H200/B300/GB300 submissions, densify B300 1k1k sweep / 刷新 H200/B300/GB300 DSv4 基准数据并加密 B300 1k1k 扫描 #2189

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Oseltamivir wants to merge 1 commit into from
Closed

[DSv4] Refresh H200/B300/GB300 submissions, densify B300 1k1k sweep / 刷新 H200/B300/GB300 DSv4 基准数据并加密 B300 1k1k 扫描 #2189

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions benchmarks/single_node/fixed_seq_len/dsv4_fp4_b300_sglang.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ start_gpu_monitor --output "$PWD/gpu_metrics.csv"
# SWA ratio: 1k inputs need more SWA cache headroom than 8k inputs; 0.5 was
# tuned empirically for the 1k1k recipe, while 0.1 is the cookbook default.

if [ "$CONC" = "1" ] || [ "$CONC" = "32" ]; then
if [ "$CONC" -le 32 ]; then
# TP-only, no DP attention
MEM_FRACTION_STATIC=0.90
SWA_FULL_TOKENS_RATIO=$([[ "$ISL" == "1024" ]] && echo 0.5 || echo 0.1)
Expand All @@ -63,7 +63,7 @@ if [ "$CONC" = "1" ] || [ "$CONC" = "32" ]; then
--disable-flashinfer-autotune
)

elif [ "$CONC" = "512" ]; then
elif [ "$CONC" -ge 64 ] && [ "$CONC" -le 512 ]; then
# DP attention, flashinfer_mxfp4
export SGLANG_OPT_SWA_EVICT_DROP_PAGE_MARGIN=1
MEM_FRACTION_STATIC=0.94
Expand Down
23 changes: 13 additions & 10 deletions configs/nvidia-master.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2002,8 +2002,8 @@ dsv4-fp4-b300-sglang:
multinode: false
# Recipes are selected inside benchmarks/single_node/dsv4_fp4_b300_sglang.sh
# by CONC:
# CONC 1|32: TP-only, flashinfer_mxfp4
# CONC 512: DP-attn, flashinfer_mxfp4
# CONC <=32: TP-only, flashinfer_mxfp4
# CONC 64-512: DP-attn, flashinfer_mxfp4
# CONC 2048-8192: DP-attn, megamoe
# ep is implicit in sglang: --moe-a2a-backend megamoe forces ep_size=tp_size,
# while low-latency leaves ep_size at the default of 1.
Expand All @@ -2014,8 +2014,12 @@ dsv4-fp4-b300-sglang:
search-space:
- { tp: 8, ep: 1, conc-start: 1, conc-end: 1 }
- { tp: 4, ep: 1, conc-start: 32, conc-end: 32 }
- { tp: 4, ep: 1, dp-attn: true, conc-start: 512, conc-end: 512 }
- { tp: 8, ep: 8, dp-attn: true, conc-start: 8192, conc-end: 8192 }
# Densified mid band: the previous single conc-512 point left the
# ~50 tok/s/user frontier region (conc 64-256) unsampled at 1k1k.
- { tp: 4, ep: 1, dp-attn: true, conc-start: 64, conc-end: 512 }
# Megamoe band extended down to the script's existing 2048/4096 profiles
# (previously only 8192 at 1k1k).
- { tp: 8, ep: 8, dp-attn: true, conc-start: 2048, conc-end: 8192 }
- isl: 8192
osl: 1024
search-space:
Expand Down Expand Up @@ -3078,7 +3082,7 @@ dsr1-fp8-h200-sglang-mtp:
# Uses the cu129 image. H200 has no FP4 path, so the FP4 indexer cache
# flag is omitted. Max-model-len is pinned at 800k per the recipe.
dsv4-fp8-h200-vllm:
image: vllm/vllm-openai:v0.21.0
image: vllm/vllm-openai:v0.25.0
model: deepseek-ai/DeepSeek-V4-Pro
model-prefix: dsv4
runner: h200
Expand All @@ -3098,11 +3102,10 @@ dsv4-fp8-h200-vllm:
- { tp: 8, ep: 1, dp-attn: false, conc-start: 1, conc-end: 256 }
- { tp: 8, ep: 8, dp-attn: true, conc-start: 1, conc-end: 256 }

# MTP variant of dsv4-fp8-h200-vllm. Uses the canonical v0.20.1 image
# (the non-MTP entry above is still on the deepseekv4-cu129 tag) and adds
# MTP variant of dsv4-fp8-h200-vllm. Mirrors the non-MTP image and adds
# --speculative-config '{"method":"mtp","num_speculative_tokens":2}'.
dsv4-fp8-h200-vllm-mtp:
image: vllm/vllm-openai:v0.21.0
image: vllm/vllm-openai:v0.25.0
model: deepseek-ai/DeepSeek-V4-Pro
model-prefix: dsv4
runner: h200
Expand Down Expand Up @@ -3143,7 +3146,7 @@ dsv4-fp8-h200-vllm-agentic:
# runner pool, search space) and adds EAGLE speculative decoding via
# --speculative-algorithm EAGLE with the (3,1,4) chain matching dsv4-fp4-b300-sglang-mtp.
dsv4-fp8-h200-sglang:
image: lmsysorg/sglang:deepseek-v4-hopper@sha256:7f19c6dc092e47a10fac2e41f47eab78970280d06648b8e50d312a82f0ae722f
image: lmsysorg/sglang:deepseek-v4-hopper@sha256:1bf5d508ab110cc0fe1659a5f21d1be02a7f0d7ca8f58cea7e7f4e11f6ae208f
model: deepseek-ai/DeepSeek-V4-Pro
model-prefix: dsv4
runner: h200-dgxc
Expand All @@ -3167,7 +3170,7 @@ dsv4-fp8-h200-sglang:
# runner pool, search space) and adds EAGLE speculative decoding via
# --speculative-algorithm EAGLE with the (3,1,4) chain matching dsv4-fp4-b300-sglang-mtp.
dsv4-fp8-h200-sglang-mtp:
image: lmsysorg/sglang:deepseek-v4-hopper@sha256:7f19c6dc092e47a10fac2e41f47eab78970280d06648b8e50d312a82f0ae722f
image: lmsysorg/sglang:deepseek-v4-hopper@sha256:1bf5d508ab110cc0fe1659a5f21d1be02a7f0d7ca8f58cea7e7f4e11f6ae208f
model: deepseek-ai/DeepSeek-V4-Pro
model-prefix: dsv4
runner: h200-dgxc
Expand Down
38 changes: 38 additions & 0 deletions perf-changelog.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4750,3 +4750,41 @@
- "Image: lmsysorg/sglang:nightly-dev-cu13-20260709-074bb928"
- "6 topologies across 1k/1k and 8k/1k: 1P1D TP4 STP + wide-EP (DEP4 prefill / DEP16 decode) from 1P1D up to 8P1D, recipes under benchmarks/multi_node/srt-slurm-recipes/sglang/qwen3.5/gb300-fp8/"
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2137

- config-keys:
- dsv4-fp8-h200-vllm
- dsv4-fp8-h200-vllm-mtp
description:
- "Bump vLLM image from v0.21.0 to v0.25.0 for DeepSeek-V4-Pro FP8 on H200, matching the B200/B300 dsv4 vLLM bump (#2169)"
- "Refresh stale H200 dsv4 submissions (last run 2026-05-21)"
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2189

- config-keys:
- dsv4-fp8-h200-sglang
- dsv4-fp8-h200-sglang-mtp
description:
- "Bump the pinned lmsysorg/sglang:deepseek-v4-hopper digest from the 2026-05-02 push (7f19c6dc) to the current 2026-05-13 push (1bf5d508)"
- "Refresh stale H200 dsv4 SGLang submissions (last run 2026-05-04)"
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2189

- config-keys:
- dsv4-fp4-b300-sglang
description:
- "Densify the 1k1k sweep: the previous search space sampled only conc {1, 32, 512, 8192}, leaving the ~50 tok/s/user frontier region unsampled and noisy (933 tok/s/GPU vs B200's 1359 at 50 tok/s/user)"
- "Add the DP-attn flashinfer_mxfp4 band conc 64-512 (was a single 512 point) and extend the megamoe band down to conc 2048-8192 (was a single 8192 point), reusing the existing CONC 2048/4096 launch profiles in dsv4_fp4_b300_sglang.sh"
- "dsv4_fp4_b300_sglang.sh: select launch profiles by CONC range (<=32 TP-only, 64-512 DP-attn flashinfer) instead of exact match; megamoe profiles unchanged"
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2189

- config-keys:
- dsv4-fp4-b300-trt
- dsv4-fp4-b300-trt-mtp
description:
- "Re-run B300 dsv4 TRT-LLM configs (no config change, last run 2026-06-11) so the 1k1k frontier density check around 50 tok/s/user compares same-pass data across frameworks"
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2189

- config-keys:
- dsv4-fp4-gb300-dynamo-trt
- dsv4-fp4-gb300-dynamo-trt-mtp
description:
- "Re-run GB300 dsv4 dynamo-trt configs (no config change): the only GB300 dsv4 1k1k submissions are from a single 2026-06-15 run"
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2189
Loading