record(ENG-CUDAGRAPH): SGLang's Breakable CUDA Graph, grounded at its pin, and the prefill lever it does not give us (#1161)

[localai-bot]

A public claim reached the operator: "SGLang is the first engine to develop and land
Breakable CUDA Graph (BCG), the full CUDA Graph, and graph memory reuse. BCG drops
torch.compile for faster setup and broader compatibility. Full graph capture brings
prefill latency down on dynamic workloads. Memory reuse keeps the graph footprint fixed
as coverage grows."

Every claim below was read in the pinned SGLang tree v0.5.15 / f63458b5be. None was
taken from the claim text. This change records the surface, the verdict, and the work
the analysis derived. No engine code is in scope.

What BCG is

BCG captures one forward as a SEQUENCE of torch.cuda.CUDAGraph segments split at
eager break points, not as one graph. eager_on_graph(True) ends the current segment,
runs the marked call eagerly once, records a replay closure over weak-ref'd args, and
begins a new segment (breakable_cuda_graph.py:204-241); replay is
for seg: seg.replay(); break_fns[i]() (:244-260); all segments share one mempool
with weak-ref'd intermediates, so the activation footprint does not grow with segment
count (:14-23,156-169). The backend says it in one line: "segment-captured graphs
with eager break markers. No torch.compile." (breakable_cuda_graph_backend.py:14-17).
The break points are the attention and recurrent calls (radix_attention.py:256,
forward_mla.py:1092, radix_linear_attention.py:159, nemotron_h.py:1240,
dsa_indexer.py:2414).

"Graph memory reuse" is two mechanisms the claim merges: that shared mempool, and
executable dedup via cudaGraphExecUpdate (cuda_graph_dedup_mixin.py:219-242, logging
"captured %d CUDA graphs, deduped to %d execs" at :358).

The priority claim is narrower than it reads

vLLM's v1 default is already FULL_AND_PIECEWISE, splitting at splitting_ops
(vllm/config/compilation.py:60-63,517,615,630 @ 555967922). The CAPABILITY has been
in the mirror source, and therefore in our benchmark denominator, all along. What SGLang
landed first is the CONSTRUCTION: piecewise coverage from runtime stream capture plus a
decorator, with no Dynamo, no Inductor, no FX. That is a real result and it removes a
compiler dependency. It is not new engine behavior, and the records now say so.

Four outcomes, three of them not throughput

(a) Dropping torch.compile is no lever here. We never had a compiler.
src/vt/cuda/cuda_backend.cu:203-232 is already raw cudaStreamBeginCapture /
cudaGraphInstantiate. BCG is SGLang arriving where a C++ engine starts.

(b) Prefill capture is REFUTED on our CUDA gate models. This is the load-bearing
result and it is recorded against ENG-CUDAGRAPH so nobody re-derives it. GB10
2026-07-09, both arms, same instrument, oracle graphed and identity-asserted:
steady-state prefill GPU-idle between launches is 3.8% with GPU-busy above 96%, and
the 27B prefill gap is 92.5% non-GEMM glue GPU work with the dominant GEMM the same
symbol at the same call count (+0.17%) and attention where we are AHEAD. There are no
launch bubbles in our prefill to collapse, and vLLM's piecewise prefill graph is already
inside the denominator we lose to. Decode is separate and already banked: concurrency-64
launch overhead was ~24% and is already one cudaGraphLaunch per step.

(c) Graph memory reuse is real and unported. cudaGraphExecUpdate appears nowhere in
src/ or include/; we instantiate one exec per padded bucket (7 at max_num_seqs=32,
11 at 64) across eight hand-rolled drivers. Filed as ENG-CUDAGRAPH-DEDUP, INVENTORIED
(#1162). Memory and capture-time, not throughput.

(d) The analysis surfaced a structural row the claim did not. Our capture is
all-or-nothing (runner.cpp:1338-1341 routes only pure_decode, so one host-dependent
op forces the whole step eager) and hand-rolled eight times. Filed as
ENG-CUDAGRAPH-BREAK, INVENTORIED (#1163). A COVERAGE row, not a throughput row, and the
records say that explicitly so it cannot be sold as speed later.

The one shape that looked like a win, and why it is blocked

Our diffusion path captures no graphs at all. SGLang measured LTX-2 two-stage H200 e2e
10.75 s → 6.90 s post-pin. But a denoise-loop graph needs a denoise loop that runs on the
device, and three open issues already measured that ours does not: GPU utilization
exactly 0 in 321 of 347 samples on a --device cuda render (#1024), no device arm
for the VAE decode (#1007), and 57-66% of wall in one resolution-constant serial host
phase (#1087). Capturing now would measure nothing. Filed as ENG-CUDAGRAPH-DIFFUSION,
INVENTORIED, blocked (#1164). The decision point is a measurement, not an
implementation.

What changed

• .agents/specs/sglang-breakable-cuda-graph.md — new spec carrying the analysis, the
  pins, the refutation, ## Owed, stop conditions and ## Outcome.
• .agents/sglang-matrix.md — new "Graph capture and runtime execution" section with the
  SGLANG-BCG row, classified INVENTORIED; rollup 44 → 45.
• .agents/engine-matrix.md — the prefill refutation recorded on ENG-CUDAGRAPH (which
  stays PARTIAL, because nothing shipped), plus the three new rows; counts updated.
• .agents/roadmap_v1.md — track C12, following the C10 precedent for folding an
  external claim.
• .agents/issue-index.md — four appended rows.

What was deliberately not done

No engine code, no capture change, no measurement taken, no oracle run, and no advance
of the SGLang pin. The two post-pin diffusion commits are cited as dated upstream events,
never as pinned evidence, and no gate here depends on them.

Analysis issue #1161, fixed in flow. Derived work is owned by #1162, #1163 and #1164,
each listed under ## Owed in the spec.

Two things the checkers corrected, kept visible

The three new rows were first filed READY and SPIKE. check-agent-record
rejected both, and it was right: READY obliges a structured spec with Scope,
Upstream chain, Our baseline, Port map, Tests to port, Gates, Dependencies, Work
breakdown and Risks/decisions, and this spec carries an analysis rather than a work
plan; SPIKE obliges a CLAIM-* owner nobody holds. Writing those sections to
satisfy a checker would have invented scope nobody measured, so all three moved to
INVENTORIED — known, not yet scoped.

ENGINE_ROWS in scripts/check-agent-record.py moves 157 to 160 for the three added
rows. That constant is the recorded mark for a counted record, not a semantic
assertion, so bumping it to match a legitimate addition is the intended maintenance
rather than a widened gate.

Gate

scripts/agent-preflight.sh on the merged tree: All gates green. — every record
gate, every mutation suite, trailer and commit-style suites, and the committed-range
now-current / doc-checkpoint / commit-trailers checks against
origin/main dd8a3b0e. test_cpu_x86_llamacpp_floor passed at load average 22.5 from
a concurrent session, so its recorded load-dependence (#618) did not fire.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [Claude Code]