[NVIDIA] DeepSeek V4 Perf Tracking

[b8zhong]

Motivation

This follows DeepSeek V4 Roadmap, which covers functional enablement; this issue is perf-only.

Scope: NVIDIA SM90 / SM10X.

---

High priority

• Integrate TRT-LLM DSv4 attention for SM100/103 [DSv4] Integrate TRT-LLM DSv4 Attention for SM100/103 #30805
• FlashInfer MNNVL backend for pure (non-fused) allreduce [NVIDIA] Add flashinfer MNNVL backend for allreduce only #30700
• Overlap scheduling for online C128 MTP [DSV4] Enable overlap scheduling for online C128 MTP #30497
• Split mixed-chunk attention: route decode tokens to the fp8 paged MLA kernel [DSV4] Split mixed-chunk attention: route decode tokens to the fp8 paged MLA kernel #30338
• Emit TMA-aligned UE8M0 scales for the FP8 einsum [DSV4] Emit TMA-aligned UE8M0 scales for FP8 einsum #34277
• Keep fp32 routing weights in the MXFP4 trtllm MoE [Deepseek V4] Keep fp32 routing weights in the mxfp4 trtllm MoE #33608 — drops the PackTopkIds launch
• Turn on Breakable Cuda Graph by default [Do Not Merge][BCG][2/N] Enable DSV4 prefill BCG: two-pass capture + DSV4-aware mem reserve #29985
• Fix performance regression caused by [Scheduler] Move the WAR barrier to right after each run_batch launch #31687
• Fused norm + RoPE + uniform fp8 store for TRT-LLM DSv4 sparse attention perf: add fused norm + RoPE + uniform fp8 store for TRT-LLM DSv4 sparse attention #32975

Attention & compression kernels

• Optimize the c128 epilogue [JIT Kernel][DSv4] Optimize epilogue of c128 #26671

Indexer & top-k

• Occupancy tuning for the DSA indexer fp8-quant Q kernel [Perf] Occupancy tuning for DSA indexer fp8-quant Q kernel #32755
• Rewrite paged_mqa_metadata perf(jit_kernel/deepseek_v4): optimize paged_mqa_metadata #25855
• Skip trivial nonpaged indexer logits [Perf] Skip trivial DSV4 nonpaged indexer logits #33857
• FlashInfer GVR top-k backend for decode Add flashinfer GVR top-k backend for DSv4 decode #33914

mHC

• FlashInfer mHC fusion feat: Add flashinfer mHC fusion for DSV4 #33616 — TileLang fusion is still faster today; this is the path when TileLang is unavailable (for future architectures), and FlashInfer should gain full pre / post+pre fusion soon

MoE & quantization

• FlashInfer MegaMoE [NVIDIA] Support flashinfer Mega Moe #31470, plus the zero-copy adapter path [MoE] Add FlashInfer MegaMOE zero-copy adapter path #33571
• Fused SwiGLU + quant for shared experts and EP-normal DeepSeek-V4 shared_expert and ep normal using fused swiglu and quant #27561
• Fused SiLU + clamp + mul + FP8-quant AOT kernel for the EP MoE path [Kernel] Add fused SiLU+clamp+mul+FP8 quant AOT kernel for DeepSeek V4 EP MoE path #32058

Speculative decoding (MTP / DSpark / EAGLE3)

• DSpark C128 online compressor [DeepSeek-V4]support dspark c128 online compressor #32035
• EAGLE3 for DeepSeek-V4-Flash-0731 [DeepSeek-V4] Support EAGLE3 for DeepSeek-V4-Flash-0731 #33344

Communication

• Fix non-EP TBO for attention TP > 1 [Bugfix] Fix DeepSeek-V4 non-EP TBO for attention TP > 1 #33250 (blocked-path guard in Fix non-EP DeepSeek-V4 TBO for attention TP > 1 #33217)
• Decode two-batch overlap with the DeepEP backend [DSV4] Support dsv4 decode TBO with deepep backend #33834
• Fix DP-attention gather semantics Fix DeepSeek-V4/DeepSeek-V4-Pro DP-attention gather semantics #31700 — dp_gather_partial treats attention-TP replicas as partial contributions

CUDA graph & scheduling

• Alt stream during BCG prefill [DSV4] perf: Enable alt stream during BCG prefill #29070
• Breakable CUDA graph for mixed-chunk prefill [WIP][DSV4] Breakable CUDA graph for mixed-chunk prefill #30420

Memory & KV capacity

• SWA recompute feat: Add DeepSeek V4 SWA recompute #31713 — keep the FULL prefix hit and rebuild only the trailing SWA window
• Decode radix cache with MTP under P/D disagg [P/D Disagg] DeepSeek-V4 decode radix cache with MTP support #31097
• BF16 C4/C128 compressed-state storage Support for C4 and C128 state storage bits BF16 on HBM space-constrained gpu(H20) #24041 — for HBM-constrained parts
• Size the SWA state pool by storage page size, not the model window [DSV4] Fix SWA state pool over-allocation by using storage page size instead of model window #30371 — recovers wasted c4-state capacity

Context Parallel

• Batched and round-robin CP prefill in the non-paged indexer [DeepSeek-V4] Support Batched and Round-Robin CP Prefill in the Non-Paged Indexer #33570
• Remove prefill CP KV and compressor materialization [Perf][DSV4] Remove prefill CP KV and compressor materialization #33236
• ag_gemm + moe_rs symmetric-memory overlap kernels for CP prefill [Perf][DSV4] add ag_gemm and moe_rs overlap kernels for dsv4 prefill #28639
• CP cache LayerSplit: common infrastructure feat: 1/N DeepSeek V4 CP Cache LayerSplit: Common Infrastructure #33382, full implementation feat: Add DeepSeek V4 CP KV LayerSplit #29187
• Decode context parallelism for DeepSeek V4 [DRAFT]add DCP support for DeepSeek V4 #30416
• Shared KV cache for prefill CP via VMM [Feat][DeepSeek V4] Shared KV Cache for Prefill CP #32059 — each CP rank owns a shard of the physical pages

Docs & recipes

• GB300 FP4 multi-node PD recipes for DeepSeek-V4-Pro Add DeepSeek-V4 Pro GB300 FP4 multi-node recipes #31377
• Slurm cluster deployment in the cookbook [Docs] Add Slurm cluster deployment to DeepSeek-V4 cookbook #30996

CI & bug tracking

• Add DSV4 NVFP4 tests [ci] Add DSV4 NVFP4 tests #28966
• Nightly AIME25 for DeepSeek-V4-Pro on B200 Add nightly AIME25 test for DeepSeek-V4-Pro on B200 #24891

---

If an open PR belongs here and I missed it, comment and I'll add it.

[zianglih]

for flashinfer fused topk api:

• feat(topk): support compact page table transforms flashinfer-ai/flashinfer#4315

[stewtong]

I have sustained access to a reserved 8x B200 SXM node and an existing 128K to 1M real-prose
benchmark and correctness harness for DeepSeek V4 (cache-busted long-context ladder, needle
checks, a 7-prompt correctness suite, decode-window TPOT arithmetic, 3-rep protocol). Recent
output from it: a controlled same-node TP8 vs DP8 comparison at 1M on V4-Flash-0731
(vllm-project/vllm#51454) and a post-#48929 MiniMax-M3-NVFP4 long-context re-run
(vllm-project/vllm#51494). The flashinfer_trtllm assertion issue in this tracker's bug
section (#26324) is also mine.

I can provide independent SM100 end-to-end validation for active perf PRs in this tracker,
on real-prose long-context workloads with correctness gates ahead of performance numbers.
Candidates I am considering, in my current order:

• [Feat][DeepSeek V4] Shared KV Cache for Prefill CP #32059 (Shared KV cache for prefill CP): the current validation is 8x H20 at concurrency 1.
  I can run 128K/512K/1M at c=1/2/4 on 8x B200 against my existing same-node TP8 and DP8
  anchors, including KV-capacity and correctness evidence.
• [NVIDIA] Support flashinfer Mega Moe #31470 / [MoE] Add FlashInfer MegaMOE zero-copy adapter path #33571 (FlashInfer MegaMoE and the zero-copy adapter): the posted benchmark is
  8-token input. I can test whether the backend crossover holds under long-context prefill
  and concurrent EP traffic.
• [Feature] Enable DeepSeek V4 IndexCache with PD, CP, and HiCache coverage #32771 (IndexCache): the posted speed tests use random token IDs. I can run real-prose
  1M with needle correctness across index_topk_freq settings.
• [Feature] Support DeepSeek V4 in PDMux #30885 (PDMux for V4): I have a published mixed-workload starvation baseline (one 1M prefill
  beside short interactive traffic) that this targets directly.

Happy to prioritize whichever would unblock upstream work first. If a PR author wants a
specific branch, config, or workload shape validated, I will run it as specified and post
raw per-request artifacts.

[stewtong]

Validation follow-up on the offer above: we completed a same-node, same-image W4A8 vs W4A4
MegaMoE A/B on DeepSeek-V4-Flash-0731 (8x B200, SGLang v0.5.17): end-to-end TTFT is unchanged
on both a scheduling-bound (dp8-attention) and a GEMM-bound (tp8, no dp-attention) shape, plus
bounded quality observations. Full results, method, and per-request artifacts offer:
#34179.

[mattteochen]

#34277

[nvpohanh]

@weireweire to share your PRs here as well.

[weireweire]

most important is finish #31687 follow up to fix it's perf regression. #32042 is my draft, but waiting for @hnyls2002 to make better fix.

another index-topk optimize: #33672

other pr from my side already included above.

[stewtong]

Shared-KV CP (#32059) validated on the new head — 2026-08-12, supersedes our 8/9 NULL verdict. Rerun on branch head 826b91a + a source rebuild of the branch aot sglang-kernel (the published 0.4.6.post1 wheel predates the shared_kv_row_cache kwarg): shared-KV CP8TP8 serves decode cleanly and is the fastest c=1 leg at every context (1M TTFT p50 61.8s vs SGLang plain-TP8 77.4s / SGLang DP8 533.5s), 27/27 ok+clean, 7/7 suite, KV 78.4M vs plain-TP8 18.9M = ~4.1x. Full detail on #32059 (issuecomment-5263951666).

Frontier note (kept open): this does NOT change the serving-topology recommendation (vLLM DP8xEP8 remains the 1M winner with ~8x KV + c>=4 win). Shared-KV was validated at c=1 only (validator pins max_running_requests=1; c>1 outside the release envelope), carries ~4.1x vs DP8's 8x replication, and the 61.8s-vs-64.9s cross-framework read is not an A/B (different checkpoint). The decisive shared-KV-vs-DP8 serving test is c>=2 with concurrent long contexts — currently excluded by the validator envelope.