SGLang landed Breakable CUDA Graph (BCG), and the whole surface is absent from
.agents/sglang-matrix.md.
ENG-CUDAGRAPH in the engine matrix is PARTIAL and carries no verdict on it either.
This issue owns recording the surface and the verdict.
What BCG is, read at our pin
Pinned SGLang tree v0.5.15 / f63458b5beaceabbd9d749b9fc956370e1b649e6, paths
relative to python/sglang/srt/.
BCG captures one forward as a sequence of torch.cuda.CUDAGraph segments split
at eager break points, not as one graph.
runner_backend_utils/breakable_cuda_graph/breakable_cuda_graph.py:204-241 —
eager_on_graph(True) wraps a function so that during capture it ends the current
segment, runs the function eagerly once, records a replay_fn closure over
weak-ref'd args, and begins a new segment.:244-260 — BreakableCUDAGraph.replay() is for seg: seg.replay(); break_fns[i]().:14-23, :156-169 — every segment shares one mempool and intermediates are
weak-ref'd, so the pool reclaims across segments and the activation footprint does
not grow with segment count.runner_backend/breakable_cuda_graph_backend.py:14-17 — "segment-captured graphs
with eager break markers. No torch.compile."
The break points are the attention and recurrent calls: layers/radix_attention.py:256,
models/deepseek_common/attention_forward_methods/forward_mla.py:1092,
layers/radix_linear_attention.py:159, models/nemotron_h.py:1240,
layers/attention/dsa/dsa_indexer.py:2414.
Graph memory reuse is two separate mechanisms: the shared mempool above, and
executable dedup at runner_backend/cuda_graph_dedup_mixin.py:219-242, which hashes
each captured graph's topology and folds compatible ones onto one cudaGraphExec
re-pointed with cudaGraphExecUpdate. It logs
"captured %d CUDA graphs, deduped to %d execs" at :358.
Backend selection is per phase: cuda_graph_config.py:38-45 offers
FULL | BREAKABLE | TC_PIECEWISE | DISABLED; decode defaults to FULL and prefill
defaults to BREAKABLE on CUDA (:95-112).
The priority claim is narrower than it reads
vLLM's v1 default is already FULL_AND_PIECEWISE — full graph for decode, piecewise
graph split at splitting_ops for prefill and mixed
(vllm/config/compilation.py:60-63,517,615,630 at our pin 555967922). The
capability has been in the mirror source, and therefore in our benchmark
denominator, for a long time. What SGLang landed first is the mechanism:
piecewise coverage from runtime stream capture plus a decorator, with no Dynamo, no
Inductor and no FX splitting. That is a real result and it removes a compiler
dependency. It is not new engine behavior, and the sglang-matrix row must say so.
The verdict this issue records
Our engine pays neither tax BCG removes. src/vt/cuda/cuda_backend.cu:203-232 is
already raw cudaStreamBeginCapture / cudaGraphInstantiate. BCG is SGLang arriving
where a C++ engine starts.
Prefill graph capture is refuted as a lever for our CUDA gate models, and this is
the part that must be recorded so nobody re-derives it. Measured on GB10 2026-07-09:
steady-state prefill GPU-idle between launches is 3.8%, GPU-busy above 96% on both
arms. The 27B prefill gap is 92.5% non-GEMM glue GPU work, with the dominant GEMM
at +0.17% and attention where we are ahead. There are no launch bubbles in our prefill
to collapse, and vLLM's piecewise prefill graph is already inside the denominator we
lose to.
Scope
- A
SGLANG-BCG row in .agents/sglang-matrix.md with the anchors above.
- The prefill-graph negative recorded against
ENG-CUDAGRAPH.
- A spec that carries the analysis and lists the derived work under
## Owed.
Not in scope: any engine code. The derived work has its own issues.
FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [Claude Code]
SGLang landed Breakable CUDA Graph (BCG), and the whole surface is absent from
.agents/sglang-matrix.md.ENG-CUDAGRAPHin the engine matrix isPARTIALand carries no verdict on it either.This issue owns recording the surface and the verdict.
What BCG is, read at our pin
Pinned SGLang tree
v0.5.15/f63458b5beaceabbd9d749b9fc956370e1b649e6, pathsrelative to
python/sglang/srt/.BCG captures one forward as a sequence of
torch.cuda.CUDAGraphsegments splitat eager break points, not as one graph.
runner_backend_utils/breakable_cuda_graph/breakable_cuda_graph.py:204-241—eager_on_graph(True)wraps a function so that during capture it ends the currentsegment, runs the function eagerly once, records a
replay_fnclosure overweak-ref'd args, and begins a new segment.
:244-260—BreakableCUDAGraph.replay()isfor seg: seg.replay(); break_fns[i]().:14-23,:156-169— every segment shares one mempool and intermediates areweak-ref'd, so the pool reclaims across segments and the activation footprint does
not grow with segment count.
runner_backend/breakable_cuda_graph_backend.py:14-17— "segment-captured graphswith eager break markers. No torch.compile."
The break points are the attention and recurrent calls:
layers/radix_attention.py:256,models/deepseek_common/attention_forward_methods/forward_mla.py:1092,layers/radix_linear_attention.py:159,models/nemotron_h.py:1240,layers/attention/dsa/dsa_indexer.py:2414.Graph memory reuse is two separate mechanisms: the shared mempool above, and
executable dedup at
runner_backend/cuda_graph_dedup_mixin.py:219-242, which hasheseach captured graph's topology and folds compatible ones onto one
cudaGraphExecre-pointed with
cudaGraphExecUpdate. It logs"captured %d CUDA graphs, deduped to %d execs"at:358.Backend selection is per phase:
cuda_graph_config.py:38-45offersFULL | BREAKABLE | TC_PIECEWISE | DISABLED; decode defaults toFULLand prefilldefaults to
BREAKABLEon CUDA (:95-112).The priority claim is narrower than it reads
vLLM's v1 default is already
FULL_AND_PIECEWISE— full graph for decode, piecewisegraph split at
splitting_opsfor prefill and mixed(
vllm/config/compilation.py:60-63,517,615,630at our pin555967922). Thecapability has been in the mirror source, and therefore in our benchmark
denominator, for a long time. What SGLang landed first is the mechanism:
piecewise coverage from runtime stream capture plus a decorator, with no Dynamo, no
Inductor and no FX splitting. That is a real result and it removes a compiler
dependency. It is not new engine behavior, and the sglang-matrix row must say so.
The verdict this issue records
Our engine pays neither tax BCG removes.
src/vt/cuda/cuda_backend.cu:203-232isalready raw
cudaStreamBeginCapture/cudaGraphInstantiate. BCG is SGLang arrivingwhere a C++ engine starts.
Prefill graph capture is refuted as a lever for our CUDA gate models, and this is
the part that must be recorded so nobody re-derives it. Measured on GB10 2026-07-09:
steady-state prefill GPU-idle between launches is 3.8%, GPU-busy above 96% on both
arms. The 27B prefill gap is 92.5% non-GEMM glue GPU work, with the dominant GEMM
at +0.17% and attention where we are ahead. There are no launch bubbles in our prefill
to collapse, and vLLM's piecewise prefill graph is already inside the denominator we
lose to.
Scope
SGLANG-BCGrow in.agents/sglang-matrix.mdwith the anchors above.ENG-CUDAGRAPH.## Owed.Not in scope: any engine code. The derived work has its own issues.
FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [Claude Code]