Repository navigation

[None][feat] Shared drafter KV pools, FP8 MLA context inputs and KV cache V2 copy/memory fixes for hybrid speculative models - #21972

Open

xguannv wants to merge 11 commits into

NVIDIA:mainfrom

xguannv:xguan/k3-rubin-runtime

Open

[None][feat] Shared drafter KV pools, FP8 MLA context inputs and KV cache V2 copy/memory fixes for hybrid speculative models#21972
xguannv wants to merge 11 commits into
NVIDIA:mainfrom
xguannv:xguan/k3-rubin-runtime

Commits on Nov 10, 2026
- [None][perf] Parallelize long KV cache block-offset copies
  
  dc3671
  authored and
  xguannv
  committed
  76a1ffe
- [None][fix] Reserve device memory for allocations outside PyTorch
  
  dc3671
  authored and
  xguannv
  committed
  0e84bd4
- [None][perf] Accept caller-quantized FP8 Q / K / V in MLA context attention
  
  dc3671
  authored and
  xguannv
  committed
  bc0cd93
- [None][feat] Share hybrid KV cache pools with speculative drafters
  
  dc3671
  authored and
  xguannv
  committed
  898663b
- [None][fix] Check a one-sided disaggregated config for its own side of shared draft KV
  xguannv
  committed
  f721369
- [None][fix] Share draft KV pages with a TRTLLM DSpark drafter only at supported page sizes
  xguannv
  committed
  197716f
- [None][fix] Scrub the shared draft KV pool at warmup in the FP4 MLA hybrid layout
  xguannv
  committed
  da67c26
- [None][chore] Shared draft KV: the drafter's declaration decides sharing in the docs; test the reverse dtype mix and the draft bytes in the hybrid estimate
  xguannv
  committed
  a7c1d27
- [None][fix] Embedded DSpark drafts skip the shared draft KV hooks
  xguannv
  committed
  029d61f
- [None][fix] Shared draft KV: the launcher turns it on only for a two-sided config, and the receiver rejects a draft layout mismatch
  xguannv
  committed
  c169570
- [None][fix] Keep the PyTorch memory cap at or above the memory PyTorch holds
  xguannv
  committed
  1fcae35