feat(ENG-CUDAGRAPH-BREAK): W4 — the persistent device input path becomes a seam capability, and it does not close the decline (#1307, #1163, #1305)

.agents/claims/CLAIM-ENG-CUDAGRAPH-BREAK-W4.md

@@ -0,0 +1,5 @@
+# CLAIM-ENG-CUDAGRAPH-BREAK-W4
+
+| Claim | Row IDs | Agent | Worktree / remote dir | Branch | Owned scope | State | Last update |
+|---|---|---|---|---|---|---|---|
+| `CLAIM-ENG-CUDAGRAPH-BREAK-W4` | `ENG-CUDAGRAPH-BREAK` (`ACTIVE`) | Claude Code (opus-5), fresh implementer | isolated worktree; one `rc` lease on `thor:gpu0` for G1; no registry, no publication | `row/ENG-CUDAGRAPH-BREAK-W4`, issue [#1307](https://github.com/mudler/vllm.cpp/issues/1307), parent [#1163](https://github.com/mudler/vllm.cpp/issues/1163), predecessors [#1192](https://github.com/mudler/vllm.cpp/issues/1192) (W1), [#1261](https://github.com/mudler/vllm.cpp/issues/1261) (W2) and [#1291](https://github.com/mudler/vllm.cpp/issues/1291) (W3) | Owns ONLY W4 of [`eng-cudagraph-break.md`](../specs/eng-cudagraph-break.md): the persistent device input path as a seam capability (`include/vt/persistent_step_input.h`, `src/vt/persistent_step_input.cpp`) with its unit gate `tests/vt/test_persistent_step_input.cpp`; the capture-faithful `Copy` in the shared `tests/vt/recording_capture_backend.h`; migrating `Qwen3_5DecodeGraph` and `Qwen3_5DenseDecodeGraph` (`src/vllm/model_executor/models/qwen3_5.cpp`) onto `vt::GraphCaptureScope` + `vt::BreakableGraph` in `kFull`; routing those drivers' `PinnedStepInputs`/`StageStepInputs` through the new capability; `vt::BreakableGraph::segment()` for the bench-only profiler marker; the gate `tests/vllm/models/test_qwen3_5_decode_graph_seam.cpp`; the two Qwen3.5 cases added to `tests/vllm/models/test_decode_graph_seam_g1_cuda.cpp`; and their CMake registration. It also owns the DISPOSITION of [#1305](https://github.com/mudler/vllm.cpp/issues/1305) and of the `qwen3.cpp` async decline ([#1179](https://github.com/mudler/vllm.cpp/issues/1179), [#323](https://github.com/mudler/vllm.cpp/issues/323)). EXCLUDES the three single-shape drivers and G5's ROCm/Tenstorrent arm (W5), the `pure_decode` predicate at `src/vllm/v1/worker/gpu/runner.cpp:1341` (W6), the auxiliary-stream auto-join (D10), and graph-executable dedup ([#1162](https://github.com/mudler/vllm.cpp/issues/1162), a parallel branch whose files this claim never touches) | `ACTIVE` | 2026-08-19 (fresh implementer) — capability landed, gated red-first and proven reached from `Qwen3_5DecodeGraph::Step`; both Qwen3.5 drivers migrated with a red-first G2 gate each, and G1 re-run and EXTENDED to them on `thor:gpu0` (5 cases, 2066 assertions, 0 differing, `4ea38eccb`); the last two batched-driver `VLLM_CPP_CUDAGRAPH` reads are gone. **The finding that changes the row's own story:** the decode graph carries NO token ids to the device in ANY driver, including the one with `StepDevInputs` — its pinned `token_ids` block was allocated, filled and never uploaded — so making that path a seam capability was never going to close the `qwen3.cpp` decline by itself. The decline therefore STANDS and #1305 stays open, both re-owned with what is actually missing named |

CMakeLists.txt

@@ -1216,6 +1216,7 @@ add_library(vllm STATIC
   src/vt/dtype.cpp
   src/vt/backend.cpp
   src/vt/breakable_graph.cpp
+  src/vt/persistent_step_input.cpp
   src/vt/tensor.cpp
   src/vt/arena.cpp
   src/vt/op_provider.cpp

docs/STATUS.md

@@ -218,8 +218,26 @@ not met.
 
 W3 (2026-08-19, #1291) migrates the three remaining plain batched drivers:
 Qwen3-Coder MoE, Voxtral text and DeepSeek-V2 MLA. Each captures and replays
-through the seam in FULL mode, and each landed with its own gate. Six drivers
-read the framework capture switch for themselves before this row; two remain.
+through the seam in FULL mode, and each landed with its own gate.
+
+W4 (2026-08-19, #1307) migrates the two Qwen3.5 drivers, so six of the nine are
+on the seam. Six drivers read the framework capture switch for themselves before
+this row; none do now, and the one read left in `src/` is the seam's own.
+
+W4 also makes the persistent device input path a shared capability rather than
+one driver's private code. `vt::PersistentStepInput` binds a capture-stable
+device destination together with its pinned host staging block, and refreshes it
+in place from a host source or a device one. The device arm is what a decode
+graph needs to read this step's real identifiers while an asynchronous scheduler
+is patching them on the GPU.
+
+Its own premise turned out to be wrong, and that is the stage's main result. The
+decode graph carries NO token ids to the device in any driver: the embedding runs
+outside the captured region from a host vector, and the one driver credited with
+the capability held a pinned token-id block that was filled every step and never
+uploaded. So the classic-dense decline that costs a shipped model its decode
+graph under asynchronous serving is not one refactor away from removable. It
+stands, and the work it needs is now named rather than assigned.
 
 W3 also closed a gate that could not fail. The mode a driver captures in was
 unobservable from outside it, so a one-token FULL-to-PIECEWISE flip left a whole

docs/USAGE.md

@@ -4420,11 +4420,15 @@ honestly, and its `total` is EXACTLY `/proc/meminfo MemTotal`
 
 ## Turning CUDA graph capture off, including the break seam
 
-`VLLM_CPP_CUDAGRAPH=0` disables CUDA graph capture. It always did for the six
-batched decode drivers that each read it, and as of `ENG-CUDAGRAPH-BREAK` W1
-(#1192) it is also the switch the shared break-point seam reads, once per
-process, into a function-local static — so a process is in exactly one lane for
-its whole life and nothing can toggle it mid-run.
+`VLLM_CPP_CUDAGRAPH=0` disables CUDA graph capture. It reached the six batched
+decode drivers as six separate reads of the same name, one copied into each
+driver; as of `ENG-CUDAGRAPH-BREAK` W4 (#1307) `src/` holds exactly ONE, in the
+shared break-point seam (`src/vt/breakable_graph.cpp`), which reads it once per
+process into a function-local static — so a process is in exactly one lane for
+its whole life and nothing can toggle it mid-run. **The switch still means what
+it always meant.** What changed is that the drivers now agree by construction
+instead of by six copies of one parse, and that the lane is fixed at the first
+read rather than re-decided whenever a driver is constructed.
 
 With capture off, or on a backend that reports no capture support (Vulkan,
 Metal, and the CPU backend), a `vt::GraphCaptureScope` is INERT: it captures
@@ -4436,14 +4440,23 @@ migration stage reversible.
 Nothing about this is new configuration to learn: there is no new flag, no new
 config key and no new command. The seam is a library surface
 (`include/vt/breakable_graph.h`), and W1 registers one break point at the dense
-attention entry of `Qwen3ForCausalLM`. No production step opens a capture scope
-yet — that arrives when the decode drivers migrate onto the seam — so today the
-switch changes nothing about the break point beyond what it already changed
-about the decode graphs.
-
-Building it needs no option. `src/vt/breakable_graph.cpp` is part of the core
-`vllm` library on every platform, because the seam is backend-agnostic and asks
-nothing new of any backend.
+attention entry of `Qwen3ForCausalLM`. **Production steps now open a capture
+scope**, six of them as of W4: `Qwen3DenseDecodeGraph` (W2, #1261),
+`Qwen3MoeDecodeGraph`, `VoxtralDecodeGraph` and `DeepseekV2DecodeGraph` (W3,
+#1291), and `Qwen3_5DecodeGraph` with `Qwen3_5DenseDecodeGraph` (W4, #1307).
+Every one of them opens the scope in FULL mode, mirroring the decode half of
+vLLM's v1 default `CUDAGraphMode.FULL_AND_PIECEWISE`, and a `vt::GraphBreak`
+inside a FULL scope takes its pass-through arm — so the switch still changes
+nothing about the break point beyond what it already changed about the decode
+graphs. This paragraph asserted the opposite until W4: it was written at W1,
+when it was true, and W2 falsified it without rewriting it here.
+
+Building it needs no option. `src/vt/breakable_graph.cpp` and, since W4,
+`src/vt/persistent_step_input.cpp` — the capture-stable per-step device input
+the migrated drivers stage through, so that a replayed graph reads this step's
+values from the address it was captured against — are part of the core `vllm`
+library on every platform, because the seam is backend-agnostic and asks nothing
+new of any backend.
 
 The switch is GATED, and it is gated in a child process, because it is read once
 per process into a function-local static and no test in a running process can

include/vt/breakable_graph.h

@@ -255,6 +255,14 @@ class BreakableGraph {
   void Replay(Queue& q);
 
   size_t segment_count() const { return segments_.size(); }
+  // THE OPAQUE HANDLE OF ONE SEGMENT, for a caller that must hand it to a
+  // profiler or a debugger. It is NOT an invitation to interpret it: spec
+  // `## Risks/decisions` D4 requires every acquisition and release to go through
+  // `Backend::EndCaptureGraph` and `Backend::DestroyGraph` so
+  // ENG-CUDAGRAPH-DEDUP (#1162) can interpose at the backend without editing
+  // this container. The one caller is the `VT_BENCH_PROFILE_CONTROL` replay
+  // marker in `qwen3_5.cpp`, which is a bench-only build.
+  void* segment(size_t i) const { return i < segments_.size() ? segments_[i] : nullptr; }
   size_t break_count() const { return break_fns_.size(); }
   bool captured() const { return !segments_.empty(); }
   int64_t replay_count() const { return replays_; }