diff --git a/.agents/engine-matrix.md b/.agents/engine-matrix.md index 839c4ac5c..0b049156d 100644 --- a/.agents/engine-matrix.md +++ b/.agents/engine-matrix.md @@ -38,7 +38,7 @@ forensics: roadmap_v1.md and the parity ledger. | Area | Rows | `ANCHOR-BACKFILL` | `PARTIAL` | `SPIKE` | `READY` | `ACTIVE` | `GATING` | `DONE` | `INVENTORIED` | |---|---:|---:|---:|---:|---:|---:|---:|---:|---:| -| Engine and scheduling | 27 | 6 | 3 | 1 | 3 | 6 | 2 | 2 | 4 | +| Engine and scheduling | 30 | 6 | 3 | 1 | 3 | 6 | 2 | 2 | 7 | | KV cache and memory | 23 | 7 | 3 | 2 | 3 | 2 | 2 | 0 | 4 | | Parallelism | 6 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 5 | | Sampling and generation | 15 | 4 | 2 | 0 | 0 | 4 | 0 | 1 | 4 | @@ -48,7 +48,7 @@ forensics: roadmap_v1.md and the parity ledger. | LoRA and adapters | 2 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 1 | | Long context and attention | 10 | 5 | 0 | 0 | 1 | 0 | 1 | 0 | 3 | | Loading, tokenizer, config | 10 | 3 | 3 | 0 | 0 | 1 | 1 | 1 | 1 | -| **Total** | **157** | **35** | **17** | **5** | **11** | **32** | **8** | **11** | **37** | +| **Total** | **160** | **35** | **17** | **5** | **11** | **32** | **8** | **11** | **40** | ## Engine core and scheduling @@ -59,7 +59,10 @@ forensics: roadmap_v1.md and the parity ledger. | `KV-PREFIX-CACHE` | APC hashes, lookup, allocation, partial blocks, eviction, plus explicit/model-default cache policy. W0 ports arbitrary-group no-prefix coordination and makes hybrid/attention-free defaults cache-off. **Full-surface re-audit 2026-07-22 ([spike](specs/prefix-prompt-caching-parity.md)) — the ported core is DEEPER than this row read (chain hashing, pool, all three coordinators, the complete hybrid intersection and four single-type managers), and the residual gaps are narrower and DIFFERENT:** **`generate_block_hash_extra_keys`: W2 DONE 2026-07-27 (`CLAIM-ROADMAP-D4APC`)** — the hardcoded no-op is replaced by a 1:1 port of `kv_cache_utils.py:451-591` (`_gen_mm_extra_hash_keys` + LoRA name + `cache_salt`, fixed order lora->mm->salt; prompt_embeds deferred, no prompt-embeds path). `Request`/`EngineCoreRequest` carry `cache_salt` + `lora_name`; `FromEngineCoreRequest` sets them BEFORE the first hash (fixed a latent ordering bug: mm_features were assigned after the ctor already hashed). The latent correctness trap is CLOSED and RED-first proven: with the stub, a tenant-B request false-hits tenant-A's 48 cached tokens (`n1==48`); with extra keys `n1==0` (no false-share). This unblocks the MM + LoRA cache consumers. **prefix-cache statistics: CLOSED 2026-07-22** (W1) — `PrefixCacheStats`/`CachingMetrics` ported 1:1 with `log_stats` DEFAULTED ON, which unblocks the `BACKEND-GATE-CUDA-SGLANG-PREFIX` hit-proof requirement; first measured hit rate 0.75 on a repeated-prefix corpus; no `cache_salt`; 1 of upstream's 4 hash algos; `skip_reading_prefix_cache` absent; partial-block primitives throw (upstream's own are DEAD CODE — no caller in `vllm/` — so they are NOT owed as live behaviour). **Also cleared: the "blocked on a supported non-hybrid family" blocker is STALE** — dense models default APC ON and five have landed, yet NO gate has ever run cache-ON **MLA prefix-cache-hit assert fixed 2026-07-23** (`CLAIM-MLA-PREFIX-CACHE-ASSERT`): `FullAttentionManager::find_longest_cache_hit` asserted `kind()==kFullAttention`, aborting DeepSeek-V2 (MLA group, kind `kMlaAttention`, APC default-ON) under asserts-enabled builds — latent since `ec6f4be`, inert under Release/NDEBUG. Relaxed to upstream's precondition `isinstance(spec, FullAttentionSpec or ChunkedLocalAttentionSpec)` (single_type_kv_cache_manager.py:578-582; MLAAttentionSpec IS-A FullAttentionSpec) ⇒ accept `kFullAttention` / `kMlaAttention` / `kChunkedLocalAttention`; restores DeepSeek-V2 SACRED gate 8/8 asserts-on, full-attention byte-identical, new MLA prefix-cache-hit unit cases. | T0 | `vllm/config/model.py:1805-1860`; `vllm/engine/arg_utils.py:510,1160-1166,2473-2508`; `vllm/config/cache.py:39,93,95`; extra keys `vllm/v1/core/kv_cache_utils.py:539-574`; hasher factory `:673-730`; `vllm/v1/core/kv_cache_coordinator.py:377-425,782-834`; `tests/v1/core/test_prefix_caching.py:225,1475,2781` | hashes/managers `src/vllm/v1/core/kv_cache_utils.cpp:259,291`; **extra_keys** `generate_block_hash_extra_keys` + `_gen_mm_extra_hash_keys` `src/vllm/v1/core/kv_cache_utils.cpp`; `cache_salt`/`lora_name` on `include/vllm/v1/request.h` + `include/vllm/v1/engine/types.h`, copied in `src/vllm/v1/request.cpp` `FromEngineCoreRequest` (fields set before the first hash); `src/vllm/v1/core/kv_cache_manager.cpp:124`; no-prefix coordinator/factory `src/vllm/v1/core/kv_cache_coordinator.cpp:260,273,279,545`; model-default/hasher selection `src/vllm/entrypoints/model_loader.cpp:109,167,180,191`; CLI `examples/server/main.cpp:126`; **statistics** `include/vllm/v1/metrics/stats.h`, recorded `src/vllm/v1/core/kv_cache_manager.cpp:139-147`, reset flag `:270-276`, take-and-swap `make_prefix_cache_stats()`, per-step window fold at the end of `Scheduler::schedule()`, accessors `Scheduler`/`EngineCore`/`LLMEngine::prefix_cache_metrics()`; `Request::num_preemptions` un-deferred (`include/vllm/v1/request.h`, incremented in `Scheduler::preempt_request`) | existing APC primitives `tests/vllm/v1/test_kv_cache_utils.cpp:411,516,536`; no-prefix hybrid allocation/no-hit `tests/vllm/v1/test_kv_cache_coordinator.cpp:213`; default/override resolution `tests/vllm/entrypoints/test_loaded_engine_dense.cpp:343`; server help and online cache-off contracts `examples/CMakeLists.txt:34`; `tests/tools/test_online_gate_client.py:582,633`; statistics plus the first MEASURED hit rate `tests/vllm/v1/test_prefix_cache_stats.cpp` 12/12; **W2 extra_keys** — ported mm/lora/salt cases + ordering + hash-level no-false-share `tests/vllm/v1/test_kv_cache_utils.cpp` (29/29), manager-level salt-partition no-false-share (RED-proven `n1 48->0`) `tests/vllm/v1/test_kv_cache_manager.cpp` (10/10), CPU gate on dgx GB10. **W3 DONE 2026-07-27 (`CLAIM-ROADMAP-D4APC-W3`, dgx GB10, NOT pushed) — the FIRST-EVER cache-ON model gate:** `tests/parity/test_qwen3_apc_e2e.cpp` on `Qwen/Qwen3-4B` (dense, full-attention, APC-default-ON) 2/2 cases, 84/84 asserts — APC-ON hits 2240/2777 (rate 0.807) / APC-OFF 0; APC-ON == APC-OFF token-exact 5/6 (1 diff a vLLM-confirmed 0.125-nat near-tie); == vLLM-APC-ON teacher-forced (OFF 6/6 gap 0.0, ON 6/6 gap ≤0.125 nats, 0 outside top-20); TTFT 70.1→39.9 ms = 1.76×. NO engine code changed (gate-only over the already-shipped default-ON path); 4B SACRED 16/16 no-regression. Oracle vLLM 0.25.0. Ledger: [parity-ledger.md#L746](parity-ledger.md#L746) | [prefix-prompt-caching-parity.md](specs/prefix-prompt-caching-parity.md) (umbrella); [prefix-caching.md](specs/prefix-caching.md) (cache-policy leaf) | `DONE` (dense APC path; W4 events/W5 partial/W6 mamba-align/W7 reset endpoint tracked in `KV-EVENTS`/`KV-MAMBA-ALIGN`/own future rows) | `a41af480` | | `KV-PREFIX-MATCH-UNIT` | `--prefix-match-unit` (config `prefix_match_unit`): the finest token boundary a prefix-cache hit can land on == the `hash_block_size`/"prefix match unit" the block hasher uses. NEW in 0.26 (absent at the prior `e24d1b24`/0.25.0 pin). For a HYBRID/multi-group model the resolver `resolve_kv_cache_block_sizes` computes `hash_block_size = prefix_match_unit if set else gcd(group_block_sizes)` (scheduler block size = `lcm`), letting matching land FINER than a physical block (e.g. 16/32 tokens inside a 1024-token block) provided every group block size is divisible by it; single-group (dense) models ignore the knob. Backs off to the scheduler block size when no prefix-cache/connector consumer is active or a mamba group diverges from `cache_block_size` (mamba_cache_mode != "align"); throws on a non-divisible unit. **W0 spike + W1 resolver LANDED 2026-07-28 (`CLAIM-PREFIX-MATCH-UNIT`, NOT pushed):** `resolve_kv_cache_block_sizes` ported 1:1 (explicit-parameter signature vs upstream's `VllmConfig`, our config surface is threaded), RED-first unit-gated (default gcd `!=` `=16` override). `PARTIAL`: the config/CLI/ABI field (W2), the scheduler threading of a resolved `hash_block_size != block_size` + mamba partial-tail stop (W3, needs the `KV-BLOCK-POOL` align path that still throws), and the benchmark (W4) are deferred. Default path byte-identical (single-group inert; scheduler still passes `block_size`). | T1 | `vllm/engine/arg_utils.py:696,1222,1940`; `vllm/config/cache.py:56-67`; resolver `vllm/v1/core/kv_cache_utils.py:626-688`; hasher `:691-748`; call site `vllm/v1/engine/core.py:154`; scheduler `vllm/v1/core/sched/scheduler.py:76,268-270,282,312-318`; fine-grained view `vllm/v1/core/single_type_kv_cache_manager.py:683,697` | resolver `src/vllm/v1/core/kv_cache_utils.cpp:638` (`resolve_kv_cache_block_sizes`), decl `include/vllm/v1/core/kv_cache_utils.h`; hash_block_size already plumbed `get_request_block_hasher` `src/vllm/v1/core/kv_cache_utils.cpp:577`; DEFERRED align path throws `src/vllm/v1/core/block_pool.cpp:93,220` (shared with `KV-BLOCK-POOL`) | `tests/vllm/v1/test_prefix_match_unit.cpp:64,88,99,119,129,145,164,186` 8/8 (29 assertions): single-group inert + DCP scale, multi-group default=gcd, `=16` override finer-than-default (RED), finer-than-1024-block, non-divisible throws, no-consumer back-off + connector-alone re-enable, mamba non-align back-off vs align gcd, hasher-granularity RED (coarse 2 vs fine 4 hashes); [parity-ledger.md](parity-ledger.md) | [prefix-match-unit.md](specs/prefix-match-unit.md) | `PARTIAL` | `CLAIM-PREFIX-MATCH-UNIT` | | `ENG-PREEMPT-RECOMPUTE` | FCFS tail preemption with recompute | T0 | `vllm/v1/core/sched/scheduler.py:1142`; `tests/v1/core/test_scheduler.py:930` | `src/vllm/v1/core/sched/scheduler.cpp:102,157`; `src/vllm/v1/core/sched/request_queue.cpp:36` | `tests/vllm/v1/test_scheduler.cpp:247,295`; `tests/vllm/v1/test_request_queue.cpp:91` | `planned: specs/preemption.md` | `ANCHOR-BACKFILL` | - | -| `ENG-CUDAGRAPH` | Decode graph capture/replay modes (host-cluster cleanup: capture-size set derived from `max_num_seqs` mirroring vLLM `_set_cudagraph_sizes`; 2026-07-18 graph-baked-scratch use-after-free fix — the 35B c2+ online-serving IMA blocker) | T0 | `vllm/config/compilation.py:53,1319,683-684,1438-1444`; `vllm/config/vllm.py:1667-1770`; `vllm/v1/worker/gpu/cudagraph_utils.py:116`; `tests/compile/test_config.py:122,229` | `src/vt/cuda/cuda_backend.cu:76,97,105`; `include/vllm/model_executor/models/decode_graph_sizes.h`; `src/vllm/model_executor/models/qwen3_5.cpp:3754,3952`; `src/vllm/v1/worker/gpu/runner.cpp:577,597`; graph-safe scratch (retire-on-grow so graph-baked scratch pointers stay valid) `src/vt/cuda/graph_safe_scratch.h`, `src/vt/cuda/cuda_moe_marlin.cu:75`, `src/vt/cuda/cuda_matmul_nvfp4.cu:766`, `src/vt/cuda/cuda_matmul_nvfp4_cutlass.cu:105`, `src/vt/cuda/cuda_matmul_fp8_cutlass.cu:95` | `tests/vt/test_cuda_backend.cpp:98`; `tests/vllm/models/test_decode_graph_sizes.cpp`; `tests/vt/test_graph_safe_scratch.cpp`; explicit 35B gate `tests/parity/test_qwen36_paged_engine.cpp:140` | [blocktable-host-cluster-cleanup.md](specs/blocktable-host-cluster-cleanup.md); [decode-graph-scratch-uaf-2026-07-18.md](specs/decode-graph-scratch-uaf-2026-07-18.md) | `PARTIAL` | - | +| `ENG-CUDAGRAPH` | Decode graph capture/replay modes (host-cluster cleanup: capture-size set derived from `max_num_seqs` mirroring vLLM `_set_cudagraph_sizes`; 2026-07-18 graph-baked-scratch use-after-free fix — the 35B c2+ online-serving IMA blocker) | T0 | `vllm/config/compilation.py:53,1319,683-684,1438-1444`; `vllm/config/vllm.py:1667-1770`; `vllm/v1/worker/gpu/cudagraph_utils.py:116`; `tests/compile/test_config.py:122,229` | `src/vt/cuda/cuda_backend.cu:76,97,105`; `include/vllm/model_executor/models/decode_graph_sizes.h`; `src/vllm/model_executor/models/qwen3_5.cpp:3754,3952`; `src/vllm/v1/worker/gpu/runner.cpp:577,597`; graph-safe scratch (retire-on-grow so graph-baked scratch pointers stay valid) `src/vt/cuda/graph_safe_scratch.h`, `src/vt/cuda/cuda_moe_marlin.cu:75`, `src/vt/cuda/cuda_matmul_nvfp4.cu:766`, `src/vt/cuda/cuda_matmul_nvfp4_cutlass.cu:105`, `src/vt/cuda/cuda_matmul_fp8_cutlass.cu:95` | `tests/vt/test_cuda_backend.cpp:98`; `tests/vllm/models/test_decode_graph_sizes.cpp`; `tests/vt/test_graph_safe_scratch.cpp`; explicit 35B gate `tests/parity/test_qwen36_paged_engine.cpp:140` | [blocktable-host-cluster-cleanup.md](specs/blocktable-host-cluster-cleanup.md); [decode-graph-scratch-uaf-2026-07-18.md](specs/decode-graph-scratch-uaf-2026-07-18.md) | `PARTIAL` | **PREFILL capture REFUTED as a lever (2026-08-17, [#1161](https://github.com/mudler/vllm.cpp/issues/1161)).** vLLM's v1 default already captures prefill piecewise (`vllm/config/compilation.py:60-63,615,630` @ `555967922`) and it is in our denominator; SGLang reached the same coverage without `torch.compile` via BCG (`SGLANG-BCG` in [sglang-matrix.md](sglang-matrix.md)). Neither helps us: GB10 2026-07-09 measured prefill GPU-idle-between-launches at **3.8%** with GPU-busy >96% on both arms, and the 27B prefill gap at **92.5% non-GEMM glue GPU work** with the dominant GEMM at +0.17% and attention AHEAD. There are no launch bubbles in our prefill to collapse. Row stays `PARTIAL`; the real residuals are exec dedup ([#1162](https://github.com/mudler/vllm.cpp/issues/1162)) and the break-point seam ([#1163](https://github.com/mudler/vllm.cpp/issues/1163)). Spec [sglang-breakable-cuda-graph.md](specs/sglang-breakable-cuda-graph.md) | +| `ENG-CUDAGRAPH-DEDUP` | Graph-executable dedup: hash each captured graph's topology and re-point ONE `cudaGraphExec` with `cudaGraphExecUpdate` on a signature hit, instead of instantiating one exec per padded bucket per model. A memory and capture-time change, NOT a throughput change — a deduped replay launches the same nodes | T2 | vLLM has no analogue (its execs come from `torch.compile`); SGLang `python/sglang/srt/model_executor/runner_backend/cuda_graph_dedup_mixin.py:219-242,358` @ `f63458b5be` | UNPORTED — `cudaGraphExecUpdate` appears nowhere in `src/` or `include/`; today `src/vt/cuda/cuda_backend.cu:222-232` instantiates a fresh exec per capture, over the 7 (`max_num_seqs=32`) or 11 (64) buckets of `include/vllm/model_executor/models/decode_graph_sizes.h:32-41`, times eight drivers | owed: same-binary A/B proving a deduped replay is byte-identical to a non-deduped one; a capture-count/exec-count log line so the ratio is observable | [sglang-breakable-cuda-graph.md](specs/sglang-breakable-cuda-graph.md) | `INVENTORIED` | [#1162](https://github.com/mudler/vllm.cpp/issues/1162) | +| `ENG-CUDAGRAPH-BREAK` | One shared `vt` capture seam that accepts BREAK POINTS, so a forward containing a host-dependent op is still graphed instead of falling out entirely — and so the eight hand-rolled drivers become one. **Coverage row, not a throughput row** | T1 | mirror vLLM `CUDAGraphMode.PIECEWISE` splitting at `splitting_ops` (`vllm/config/compilation.py:60-63,517,615,630` @ `555967922`); construction from SGLang BCG `breakable_cuda_graph.py:204-241,244-260` @ `f63458b5be` (decorator + runtime stream capture, no compiler) | all-or-nothing today: `src/vllm/v1/worker/gpu/runner.cpp:1338-1341` routes only `pure_decode`; drivers `qwen3_5.h:275`, `qwen3_5_dense.h:391`, `qwen3_moe.h:117`, `qwen3.h:243`, `deepseek_v2.h:324`, `voxtral.h:126`, plus `deepseek_v4.cpp`, `laguna.cpp` | owed: reachability mutation (delete the production call site, rerun the focused gate); bit-exactness vs eager on every migrated model over MORE than one replay; the host-lifetime contract of [decode-graph-scratch-uaf-2026-07-18.md](specs/decode-graph-scratch-uaf-2026-07-18.md) enforced AT the seam | [sglang-breakable-cuda-graph.md](specs/sglang-breakable-cuda-graph.md) | `INVENTORIED` | [#1163](https://github.com/mudler/vllm.cpp/issues/1163), [#1020](https://github.com/mudler/vllm.cpp/issues/1020) | +| `ENG-CUDAGRAPH-DIFFUSION` | Capture the LTX-2.5 denoise loop (fixed shapes, many identical iterations — the ideal graph target). **BLOCKED, and the blocker is ours:** the render does almost no device compute to capture | T2 | SGLang enabled BCG on this shape AFTER our pin — LTX-2 H200 two-stage 10.75s->6.90s (`d4be483efb`), SANA 1024px -26% (`6c7498113f`), SANA denoise 0.73->0.457s (`56ef810cad`). Dated events, NOT pinned evidence; their win is mostly PyTorch host tax we do not pay | NO capture at all: `grep` for capture across `src/vllm/model_executor/models/ltx2*.cpp` returns nothing | blocked by [#1024](https://github.com/mudler/vllm.cpp/issues/1024) (GPU util **exactly 0 in 321 of 347 samples**, 1.00 core of 20 held for 17+ min after staging), [#1007](https://github.com/mudler/vllm.cpp/issues/1007) (VAE decode has no device arm), [#1087](https://github.com/mudler/vllm.cpp/issues/1087) (**57-66% of wall** is ONE resolution-CONSTANT serial host phase), [#1010](https://github.com/mudler/vllm.cpp/issues/1010) (no phase-boundary log). Decision point is a MEASUREMENT of GPU-busy vs wall once device-resident, not an implementation | [sglang-breakable-cuda-graph.md](specs/sglang-breakable-cuda-graph.md) | `INVENTORIED` | [#1164](https://github.com/mudler/vllm.cpp/issues/1164) | | `ENG-BATCH-INVARIANT` | Opt-in deterministic execution across scheduler batch sizes (`VLLM_BATCH_INVARIANT=1`): batch-invariant matmul/norm/attention/collectives plus persistent-scheduler NVFP4; production default remains off | T1 | default/env `vllm/envs.py:89,576-578`; initialization `vllm/v1/worker/gpu_worker.py:1262`; NVFP4 dispatch `csrc/libtorch_stable/quantization/fp4/nvfp4_scaled_mm_sm120_kernels.cu:212-220`; suite fixture `tests/v1/determinism/conftest.py:9-12`; operator/e2e `tests/v1/determinism/test_nvfp4_batch_invariant_scaled_mm.py`, `tests/v1/determinism/test_nvfp4_batch_invariant.py` @ `702f481` | - | [W3-C3R executed contract](specs/nvfp4-persistent-plan-cache.md#w3-c3r-batch-shape-localization-and-gate-correction-2026-07-13): production-default ours and vLLM both change outputs across batch shapes; no local opt-in implementation is claimed | `planned: specs/batch-invariant-execution.md` | `INVENTORIED` | - | | `ENG-ASYNC-SCHED` | Async/overlap scheduling (AsyncScheduler placeholders + depth-2 batch-queue step + async D2H on a copy stream); vLLM's DEFAULT at the pin — mirror obligation per B3. **Host-side machinery + runner device-input half + sampler-OUTPUT half LANDED + CPU-gated (2026-07-16):** `AsyncScheduler` placeholder accounting, `step_with_batch_queue` depth-2, `ResolveAsyncScheduling` default-ON-when-compatible + `MaxConcurrentBatches`, `VT_ASYNC_SCHED` rollback; the runner device-input path `combine_sampled_and_draft_tokens`; PLUS the sampler-OUTPUT half — `vt::Backend` event/pinned primitives (`AllocPinned`/events, CUDA cudaHostAlloc+cudaEvent, CPU sync-degeneration), `AsyncGPUModelRunnerOutput` (device sampled-id snapshot → non-blocking D2H on a copy queue + event; `get_output()` waits only that event; MAIN queue never blocked), `Sampler::forward(sampled_ids_out)` device-resident greedy, `GPUModelRunner::sample_tokens_async` + `runner_supports_async`, and the `Executor`+`step_with_batch_queue` seam resolving `get_output()` at CONSUME time. All behind `VT_ASYNC_RUNNER`/`set_async_input_combine`, default OFF. Sync path byte-identical (placeholder sites INERT while count 0; combine off; `sample_tokens_async` degenerates to sync when async off; `sampled_ids_out=nullptr`). **ENABLE-FLIP LANDED + CPU-gated (2026-07-16):** (1) `LoadedEngine` now reorders `runner_` before the scheduler and builds an `AsyncScheduler` + `max_concurrent_batches=2` when `ResolveAsyncScheduling(runner_.runner_supports_async())` resolves ON (else the byte-identical synchronous `Scheduler` + depth-1); the resolved mcb threads into `AsyncLLM`→`EngineCoreProc` (`step_with_batch_queue`) and the "Asynchronous scheduling is enabled/disabled" log mirrors vLLM for A/B audit; (2) the device combine/scatter kernel (`_combine_sampled_and_draft_tokens_kernel` + last_sampled scatter) is ported to CUDA (`src/vt/cuda/cuda_combine_tokens.cu`), main-stream-ordered on the CUDA async path so it DELETES `sample_tokens_async`'s pre-scatter `Synchronize`; the CPU backend keeps the host loop. `VT_ASYNC_RUNNER=1` engages full W3; `VT_ASYNC_SCHED=0` is the same-binary rollback. Production default (no env) stays synchronous byte-identical. **FULL W3 DGX proof RAN twice** — `f086b64` (5/5 gates PASS; c16 TPOT −5.4 ms WIN, tput neutral, TTFT +36 % = Little's-law repayment) and the 2026-07-16 re-proof on the THROUGHPUT-lever fix (persistent pooled sampled-id/pinned buffers + `Sampler` greedy scratch removing ALL per-step `cudaMalloc`/`cudaFree`/`cudaHostAlloc`/event-create from the sampled-id path, incl. the overlap-killing `cudaFree` inside `get_output`; mirrors `gpu_model_runner.py:873-878` + `async_utils.py:12-70`): token-exactness **6/6 PASS**, interleaved c16 **tput −0.32 % (gate ≥+1.5 % FAILS), TPOT −4.95 ms retained, TTFT +34.8 %** — the allocator lever is REFUTED as the tput unlock (≤0.1 % of a ~165 ms c16 step). **DEFAULT FLIPPED ON 2026-07-17** (`VT_ASYNC_RUNNER` default ON via the pure `AsyncRunnerFlagIsOn` predicate, mirroring `vllm/config/vllm.py:992-1044`): the discriminator (`6ea7856`) proved vLLM's own async pays the identical +26–31 % TTFT / −0.7 to −0.9 % tput / −2.6 to −4.3 ms TPOT envelope and W3-ON nets positive (both binding ITL-tail anomalies flip to PASS), so the "needs a throughput lever" ship-gate is RETIRED — W3 is a parity/mirror obligation with a tails+TPOT win. The flip is TOKEN-NEUTRAL (async-ON ≡ async-OFF bit-identical on DGX). `VT_ASYNC_RUNNER=0` = runner-level rollback, `VT_ASYNC_SCHED=0` = scheduler-level rollback. TTFT means rise into vLLM's async envelope BY DESIGN — the next binding grid runs async by default and its TTFT must NOT be misread as a regression. **ROBUSTNESS FIX 2026-07-20 (`discard_request_mask`):** the runner was missing vLLM's `discard_request_mask`, so `GPUModelRunner` emitted a sampled token for prefill-CHUNK requests too; under async this drained a `num_output_placeholders` never reserved (the `is_prefill_chunk` path adds none) → the `async_scheduler.cpp` `num_output_placeholders >= 0` assertion aborted on c8 + short-output (chunked prefill + preemption). FIX mirrors vLLM: `execute_model` computes `exec_state_.discard[i] = seq_len < num_tokens` (`gpu_model_runner.py:2048`); `sample_tokens` clears those rows to empty (`outputs.py:303`), the async path passes `invalid_req_indices` to `AsyncGPUModelRunnerOutput::get_output` (`gpu_model_runner.py:3625` + `outputs.py:303`). Scheduler UNCHANGED (assertion kept — it was correct once the runner honors `scheduler.py:1888-1890`). Sync/non-chunked decode byte-identical (mask all-zero); DGX 27B 235/235 + 35B 315/315, `vllm-bench` c8+short-output+chunked+kv-pressure no longer crashes, memcheck 0. Ledger [parity-ledger.md](parity-ledger.md) 2026-07-20 row | T1 | `vllm/v1/core/sched/async_scheduler.py:12`; `vllm/config/vllm.py:490,990,1038`; `vllm/v1/engine/core.py:519`; `vllm/v1/worker/gpu/input_batch.py:304-406`; `vllm/v1/worker/gpu/async_utils.py:12-70`; `vllm/v1/worker/gpu/gpu_model_runner.py:242-332`; `vllm/v1/outputs.py:298-307` | `src/vllm/v1/core/sched/async_scheduler.cpp:10,45`; placeholder plumbing `src/vllm/v1/core/sched/scheduler.cpp:148,164,605`; `src/vllm/v1/engine/core.cpp:91` (`step_with_batch_queue`, async-output seam); `src/vllm/v1/engine/core_proc.cpp:32,46`; config `include/vllm/config/scheduler.h:117,165,188`, `src/vllm/config/scheduler.cpp:12`; `include/vllm/v1/request.h:187`; runner input leaf `src/vllm/v1/worker/gpu/prepare_inputs.cpp`, `src/vllm/v1/worker/gpu/input_batch.cpp`; runner output leaf `include/vt/backend.h`+`src/vt/backend.cpp`+`src/vt/cuda/cuda_backend.cu` (event/pinned), `include/vllm/v1/worker/gpu/async_output.{h,cpp}` (`AsyncGPUModelRunnerOutput`), `src/vllm/v1/sample/sampler.cpp` (`sampled_ids_out`), `src/vllm/v1/worker/gpu/runner.cpp` (`sample_tokens_async`/`runner_supports_async`), `src/vllm/v1/executor/executor.cpp`+`include/vllm/v1/worker/gpu/model_runner_base.h` (async seam); enable-flip `include/vllm/entrypoints/model_loader.h`+`src/vllm/entrypoints/model_loader.cpp` (`runner_` before scheduler, `ResolveAsyncEnabled`/`MakeScheduler`, `AsyncScheduler`+mcb=2, log), `include/vllm/v1/engine/async_llm.h`+`src/vllm/v1/engine/async_llm.cpp` (mcb param → `EngineCoreProc`); device kernel `include/vt/cuda/combine_tokens.h`+`src/vt/cuda/cuda_combine_tokens.cu`, wired `src/vllm/v1/worker/gpu/runner.cpp` (CUDA combine/scatter branch removes the pre-sync) | `tests/vllm/v1/test_async_scheduler.cpp:1` (6 cases, 54 asserts; RED vs base Scheduler 2/6 fail); depth-2 engine cycle `tests/vllm/v1/test_engine_core_proc.cpp:479` (mcb=2, async-output seam); config resolution `tests/vllm/test_scheduler_config.cpp:75`; enable-flip construction matrix `tests/vllm/entrypoints/test_loaded_engine_dense.cpp` (runner×VT_ASYNC_SCHED → scheduler type + mcb; RED = un-flipped engine, 3/3 ON-arm asserts fail); runner input leaf `test_combine_tokens.cpp` (RED = stale → 5/7 fail), `test_input_batch.cpp`, `test_runner.cpp` (async-ON≡sync); output leaf `tests/vt/test_backend.cpp` (event/pinned contract), `tests/vllm/v1/worker/test_async_output.cpp` (materialize/flush/snapshot; RED = +1 splice), `test_runner.cpp` (`sample_tokens_async` decode ≡ sync); full CPU ctest 111/111, tools 164/164. Prior diagnostic `3812d8` six-leg control: total **1.002153×**, TTFT **0.862159×**, no GPU-time reduction (neutral for speed). **DEFAULT-FLIP (2026-07-17):** new pure CPU flag test [test_async_runner_flag.cpp](../tests/vllm/v1/worker/test_async_runner_flag.cpp) (11 asserts, default-ON/'0'-off); construction matrix [test_loaded_engine_dense.cpp](../tests/vllm/entrypoints/test_loaded_engine_dense.cpp) INVERTED (default → AsyncScheduler+mcb=2; RED verified 5 asserts fail vs un-flipped). CPU clean `-Werror` rebuild, full serial ctest **116/116**, tools **164/164**. **DGX re-confirmation** (evidence `dgx:~/work/vllm.cpp-async-flip`, CUTLASS+FA2 hard-verified, one flock): shipping default (async ON + RMSNorm-fast OFF) → **27B 235/235 + 35B 315/315** with the "Asynchronous scheduling is enabled (mcb=2)" log, and both rollback arms (`VT_ASYNC_RUNNER=0`, `VT_ASYNC_SCHED=0`) 235/235 + 315/315 log "disabled"; async arms BIT-IDENTICAL (token-neutral). Closing record [parity-ledger.md#L502](parity-ledger.md#L502) | [async-serving.md](specs/async-serving.md) | `DONE` | `6ea7856` | | `ENG-PRIORITY-SCHED` | Priority request queue + policy + priority preemption + `priority` plumbing (Request/EngineCoreRequest/OpenAI field); W4 of the async-serving block. Default stays FCFS. GATING: full CPU tier green (93/93; 12 ported priority-scheduler cases + 14 priority-queue cases incl. the seeded random property test); GPU G1 (both greedy engine gates, priority-vs-fcfs token-exactness) deferred to the next GPU-idle window — GPU held by the `SERVE-GATE-ONLINE` campaign | T1 | `vllm/v1/core/sched/request_queue.py:131,201`; `vllm/v1/core/sched/scheduler.py:546`; `vllm/config/scheduler.py:109`; `tests/v1/core/test_scheduler.py:2382,2978`; `tests/v1/core/test_priority_scheduler_random.py:1` | `src/vllm/v1/core/sched/request_queue.cpp:101,186`; `src/vllm/v1/core/sched/scheduler.cpp:178`; `src/vllm/v1/request.cpp:92`; `src/vllm/config/scheduler.cpp:21` | `tests/vllm/v1/test_scheduler.cpp:674,916`; `tests/vllm/v1/test_request_queue.cpp:238,429` | [async-serving.md](specs/async-serving.md) | `GATING` | - | diff --git a/.agents/issue-index.md b/.agents/issue-index.md index 295c2d1d8..a4cdd3254 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -347,3 +347,7 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#1150](https://github.com/mudler/vllm.cpp/issues/1150) | `LTX25-TI2VID-RECIPE` | The sigma SHIFT is derived from the target latent on every arm, where six of upstream's seven `LTX2Scheduler.execute` call sites pass NO latent and take `default_number_of_tokens` = `MAX_SHIFT_ANCHOR` = 4096 (`schedulers.py:11,:29,:31`). `grep -rn '\.execute(' packages/ltx-pipelines/src/ltx_pipelines/` at `fd4ded7f` returns seven and that grep is the whole population: only `ti2vid_two_stages_hq.py:267` passes `latent=empty_latent`. This engine passes `target_tokens` at `src/vllm/multimodal/ltx2_video.cpp:3442-3443`, so it mirrors the exception and diverges from the rule. Correct today: `t2a_one_stage` (passes 0 at `src/vllm/model_executor/models/ltx2_t2a.cpp:178`) and `res2s_two_stage`. DIVERGENT: `one_stage` at four version keys (`ti2vid_one_stage.py:207`), `a2vid_two_stage` stage 1 (`a2vid_two_stage.py:226`) and `retake`'s non-distilled arm (`retake.py:287`). Recipes carrying explicit `sigmas` never reach the derivation and are unaffected. Not a rounding difference: at the recipe default geometry the target latent is 6144 tokens, giving `sigma_shift` 2.78 against upstream's 2.05, so every sigma moves. Invisible because the trajectory changes while the frame count, shapes, sample rate and errors do not, and our goldens were captured from this engine so they PIN it rather than detect it. `.agents/specs/ltx25-res2s-loop.md:80-88` saw the HQ/plain split and concluded the divergence was on the plain two-stage arm alone; that is right about HQ and wrong about the blast radius. Found by row `LTX25-TI2VID-RECIPE`, which added the seam — `Ltx2PhaseRecipe::schedule_tokens`, defaulted to today's behaviour so nothing moves — and set it on the one phase it ships. Not fixed in flow because flipping the other three re-samples five shipped, gated arms and rewrites their goldens, which needs its own spec and fresh review. Listed under `## Owed` in [`ltx25-ti2vid-recipe.md`](specs/ltx25-ti2vid-recipe.md) | bug | | [#1151](https://github.com/mudler/vllm.cpp/issues/1151) | `LTX25-TI2VID-RECIPE` | The `requires_distilled_lora` refusal still advertised [#1118](https://github.com/mudler/vllm.cpp/issues/1118) as OPEN, and cited `a2vid_two_stage.py`'s line numbers to every other pipeline. #1118 closed at `4ae0f54ab` (row `LTX25-PHASE-LORA`, PR [#1140](https://github.com/mudler/vllm.cpp/pull/1140)), which added `Ltx2PhaseRecipe::loras` and `Ltx2RebindDitLoras`; the message at `src/vllm/multimodal/ltx2_video.cpp:1039-1041` still ended "upstream fuses that adapter into stage 2 ALONE and this engine fuses once at load, so stage 1 sees it too", every clause of which had become false, and the comment above the refusal said the same. `ltx25-phase-lora.md` repaired the REFERENCE-CONDITIONING refusal, which carried the identical claim ~1100 lines away, and named only that one in its port map, so this site and the `ltx2-gen --help` text (`examples/ltx2_gen/main.cpp:210-212`) were both missed. Second defect at the same site: the refusal is deliberately keyed on the FLAG rather than on the kind string so the next recipe inherits it (the comment names #1093 and #1096 as waiting), yet its body interpolated `im.pipeline_kind` into the first sentence and hard-coded a2vid's `:164`, `:114`, `:107` into the rest — so the first arm to inherit it would be told its own name and then a different pipeline's source lines. `--distilled-lora required=True` lives on `default_2_stage_arg_parser` (`utils/args.py:1123`, `:1140-1155`), which all of these pipelines select, and that shared anchor is what the message now cites. Found and fixed IN FLOW by row `LTX25-TI2VID-RECIPE`, the second user of the flag; `tests/vllm/multimodal/test_ltx2_video.cpp` asserted the string `1118` was PRESENT and now asserts it is absent | bug | | [#1152](https://github.com/mudler/vllm.cpp/issues/1152) | `LTX25-TI2VID-RECIPE` | `Ltx2PipelineRecipe::allow_request_latents` (`include/vllm/model_executor/models/ltx2_pipeline.h:705`) is WRITTEN by every recipe and READ by nothing — the "a parameter no caller passes" shape AGENTS.md `## Nothing lands dead` names. Measured at `c83b96934`: `grep -rn allow_request_latents src include examples` minus the declaration returns FIVE lines and all five are assignments (`ltx2_pipeline.cpp:1264` false, `:1345` false, `:1473` true, `:1604` false). Positive control, the field declared one line above and set in the same blocks: `allow_request_sigmas` returns its assignments PLUS a real reader at `src/vllm/multimodal/ltx2_video.cpp:3476`, so the grep is well-formed and the absence is the finding. Consequence: `Res2sTwoStageRecipe` carries `true` where every other recipe carries `false` and nothing can tell the difference — no upstream `__call__` among these pipelines takes an initial-latent parameter, so `false` is what the signatures support and the `true` looks like an oversight, but it is unfalsifiable while nothing reads the field. The tests assert the VALUES, so they gate the record against itself and cannot see that nothing consumes it, which is the tautology shape [#911](https://github.com/mudler/vllm.cpp/issues/911) recorded on the anchor checker. Two closes: give it a reader (a refusal on a request supplying a latent to a recipe whose upstream signature has none) or delete it and its assertions — deleting is defensible, since no request surface carries a latent at all so the refusal could never fire either. Found by row `LTX25-TI2VID-RECIPE` while deriving the same field for a sixth recipe; not fixed in flow because both closes touch five landed recipes and one deletes gated assertions. Listed under `## Owed` in [`ltx25-ti2vid-recipe.md`](specs/ltx25-ti2vid-recipe.md) | bug | +| [#1161](https://github.com/mudler/vllm.cpp/issues/1161) | `ENG-CUDAGRAPH` | SGLang landed **Breakable CUDA Graph (BCG)** and the whole surface was absent from `.agents/sglang-matrix.md`, while `ENG-CUDAGRAPH` carried no verdict on it. Grounded at pin `v0.5.15`/`f63458b5be`: BCG captures one forward as a SEQUENCE of `torch.cuda.CUDAGraph` segments split at eager break points on the attention and recurrent calls, sharing one mempool with weak-ref'd intermediates, with **no `torch.compile`** (`breakable_cuda_graph.py:204-241,244-260,14-23,156-169`; `breakable_cuda_graph_backend.py:14-17`; break points `radix_attention.py:256`, `forward_mla.py:1092`, `radix_linear_attention.py:159`, `nemotron_h.py:1240`, `dsa_indexer.py:2414`). **The public priority claim is narrower than it reads:** vLLM's v1 default is ALREADY `FULL_AND_PIECEWISE`, splitting at `splitting_ops` (`vllm/config/compilation.py:60-63,517,615,630` @ `555967922`), so the capability has been in our benchmark denominator all along — what SGLang landed first is the CONSTRUCTION (piecewise coverage from runtime stream capture plus a decorator, no Dynamo/Inductor/FX), which matters to us only because we have no compiler either. **The load-bearing outcome is a REFUTATION:** prefill graph capture is not a lever on our CUDA gate models, because GB10 2026-07-09 measured steady-state prefill GPU-idle-between-launches at **3.8%** with GPU-busy >96% on both arms, and attributed the 27B prefill gap to **92.5% non-GEMM glue GPU work** with the dominant GEMM the same symbol at the same call count (+0.17%) and attention AHEAD. There are no launch bubbles in our prefill for a graph to collapse, and vLLM's piecewise prefill graph is already inside the denominator we lose to, so it does not explain the gap either. Decode is a separate and already-banked case (conc-64 launch overhead was ~24%, already one `cudaGraphLaunch`/step). FIXED IN FLOW: `SGLANG-BCG` row added under a new "Graph capture and runtime execution" section of `sglang-matrix.md`, the refutation recorded on `ENG-CUDAGRAPH` (which stays `PARTIAL`, because nothing shipped), roadmap track `C12` added, and spec [`sglang-breakable-cuda-graph.md`](specs/sglang-breakable-cuda-graph.md) written to carry the analysis. NOT fixed here and owned by the three issues below: no engine code, no measurement taken, no oracle run, and no advance of the SGLang pin | record | +| [#1162](https://github.com/mudler/vllm.cpp/issues/1162) | `ENG-CUDAGRAPH-DEDUP` | We instantiate **one `cudaGraphExec` per padded decode bucket, per model**, and `grep -rn "cudaGraphExecUpdate" src include` returns nothing. `src/vt/cuda/cuda_backend.cu:222-232` instantiates a fresh exec per capture; `include/vllm/model_executor/models/decode_graph_sizes.h:32-41` yields 7 buckets at `max_num_seqs=32` and 11 at 64; eight drivers each build their own set. SGLang folds compatible captures onto one executable by hashing graph topology and calling `cudaGraphExecUpdate` on a signature hit (`cuda_graph_dedup_mixin.py:219-242`, logging "captured %d CUDA graphs, deduped to %d execs" at `:358`). Portable to us unchanged, because it is driver-level rather than PyTorch-level. It is a **memory and capture-time** change, NOT a throughput change — a deduped replay launches the same nodes — and it matters because on GB10 unified memory an OOM reboots the box, capture time is startup latency (a recorded gate axis), and bucket count is exactly what widening graph coverage would raise. Owed: the dedup registry behind the `vt` seam, a capture-count/exec-count log line, and a same-binary A/B proving a deduped replay is byte-identical rather than asserting it. Hazard already recorded: capture bakes host source addresses and a clean `compute-sanitizer` run is NOT evidence a capture path is safe (`specs/decode-graph-scratch-uaf-2026-07-18.md`). Spec [`sglang-breakable-cuda-graph.md`](specs/sglang-breakable-cuda-graph.md) `## Owed`. Analysis: [#1161](https://github.com/mudler/vllm.cpp/issues/1161) | perf | +| [#1163](https://github.com/mudler/vllm.cpp/issues/1163) | `ENG-CUDAGRAPH-BREAK` | Two defects, stated separately. **(1) Capture has no break points:** `src/vllm/v1/worker/gpu/runner.cpp:1338-1341` routes only `pure_decode` batches to a graph, so prefill, mixed batches, and anything whose metadata is computed on the host stay eager for the WHOLE step. There is no way to express "capture this forward except for these three calls", so coverage is a cliff rather than a slope; [#1020](https://github.com/mudler/vllm.cpp/issues/1020) is one instance (a spec verify step whose actual draft depth differs from the configured `k` silently falls out to eager). **(2) Eight hand-rolled drivers:** `Qwen3_5DecodeGraph` (`qwen3_5.h:275`), `Qwen3_5DenseDecodeGraph` (`qwen3_5_dense.h:391`), `Qwen3MoeDecodeGraph` (`qwen3_moe.h:117`), `Qwen3DenseDecodeGraph` (`qwen3.h:243`), `DeepseekV2DecodeGraph` (`deepseek_v2.h:324`), `VoxtralDecodeGraph` (`voxtral.h:126`), plus graph code in `deepseek_v4.cpp` and `laguna.cpp` — each re-deriving capture, bucket padding, persistent-input threading and the pure-decode predicate. AGENTS.md names this shape: a parallel path written by hand instead of a shared seam, and every new model that wants a decode graph writes a ninth one. Mirror vLLM's `CUDAGraphMode.PIECEWISE` boundary (`splitting_ops`, i.e. the attention ops); take the CONSTRUCTION from SGLang BCG, because vLLM gets its split from `torch.compile` and we have no compiler. **Explicitly NOT a throughput row and must not be sold as one** — our prefill has 3.8% host idle and is >96% GPU-busy, and decode already banked its launch-overhead win; the value is coverage and one seam instead of eight. Any speed claim must first name and measure a path that is currently eager AND currently host-bound. Owed: the seam, break-point registration, a reachability mutation (delete the production call site, rerun the focused gate), and bit-exactness vs eager on every migrated model over MORE than one replay. Large and structural; spike first. Spec [`sglang-breakable-cuda-graph.md`](specs/sglang-breakable-cuda-graph.md) `## Owed`. Analysis: [#1161](https://github.com/mudler/vllm.cpp/issues/1161) | feature | +| [#1164](https://github.com/mudler/vllm.cpp/issues/1164) | `ENG-CUDAGRAPH-DIFFUSION` | Our diffusion path captures **no CUDA graphs at all** — `grep` for capture across `src/vllm/model_executor/models/ltx2*.cpp` returns nothing — and a denoise loop is the ideal capture target (fixed shapes, tens of identical iterations). SGLang enabled BCG on exactly this shape AFTER our pin and measured LTX-2 two-stage H200 e2e 10.75 s → 6.90 s (`d4be483efb`), SANA 1024px e2e -26% (`6c7498113f`), SANA denoise 0.73 → 0.457 s (`56ef810cad`), and Z-Image BCG made bit-exact vs eager (`f5f0c3ee7a`) — cited as DATED UPSTREAM EVENTS, never as pinned evidence, and to be discounted hard because their win is mostly PyTorch host tax our C++ loop does not pay. **BLOCKED, and the blocker is ours:** the premise a denoise-loop graph needs is a denoise loop that RUNS ON THE DEVICE, and three open issues already measured that ours does not. [#1024](https://github.com/mudler/vllm.cpp/issues/1024): a `--device cuda` render stages 35.54 GiB onto the GPU and then does no compute on it — GPU utilization **exactly 0 in 321 of 347 samples**, all 26 non-zero samples inside the staging window, and from t=251 s onward over 17 minutes of 0 while the process holds exactly 1.00 core of 20. [#1007](https://github.com/mudler/vllm.cpp/issues/1007): the video VAE decode has no device arm. [#1087](https://github.com/mudler/vllm.cpp/issues/1087): after [#1041](https://github.com/mudler/vllm.cpp/issues/1041), **57-66% of render wall** is one contiguous single-threaded phase measuring 1731 s and 1732 s across two rungs whose voxel counts differ 2.75x, i.e. resolution-CONSTANT, and [#1010](https://github.com/mudler/vllm.cpp/issues/1010) is open because nothing timestamps a phase boundary so the phase is not even named. A graph collapses host launch dispatch; there is close to no device dispatch here to collapse and the dominant cost is a serial host phase a graph cannot touch, so **capturing now would measure nothing**. Unblock order: [#1010](https://github.com/mudler/vllm.cpp/issues/1010) phase log → [#1087](https://github.com/mudler/vllm.cpp/issues/1087) name and fix the serial phase → [#1024](https://github.com/mudler/vllm.cpp/issues/1024)/[#1007](https://github.com/mudler/vllm.cpp/issues/1007) device residency → THEN measure GPU-busy vs wall. That fourth step is the decision point and it is a MEASUREMENT, not an implementation: if the loop is device-resident and host-dispatch-bound, capture it; if GPU-bound, close it the way [#1161](https://github.com/mudler/vllm.cpp/issues/1161) closed prefill. Gate when it runs: bit-exact PIXELS vs eager over more than one replay, never an exit code (see [#1149](https://github.com/mudler/vllm.cpp/issues/1149)). Spec [`sglang-breakable-cuda-graph.md`](specs/sglang-breakable-cuda-graph.md) `## Owed` | perf | diff --git a/.agents/roadmap_v1.md b/.agents/roadmap_v1.md index a9c7f870f..879db5a67 100644 --- a/.agents/roadmap_v1.md +++ b/.agents/roadmap_v1.md @@ -149,6 +149,7 @@ live in [feature-matrix.md](feature-matrix.md). | C9 | Recurring upstream sync cycle and P1 sync tooling | 🔁 recurring; [upstream-sync.md](upstream-sync.md) | | C10 | **Qwen3.5 high-throughput lever set** (user-directed 2026-08-08, from [vllm.ai 2026-08-06 "25K tok/s/GPU"](https://vllm.ai/blog/2026-08-06-qwen35-25k-tps)): mirror the three levers upstream credits. **(a) Blackwell GDN prefill** replacing the FLA/Triton kernel (FlashInfer PR #3001, vLLM PR #40717, `--gdn-prefill-backend flashinfer`): 1.02x-5.78x on the kernel across sizes, 5.92x microbenchmark, **1.13x e2e prefill throughput, 12% lower mean TTFT**. **(b) Hybrid cache + GDN-state transfer** (vLLM PR #36687 + #37416/#37635/#37310/#41869, `VLLM_SSM_CONV_STATE_LAYOUT=DS`): ~7% intra-node H100 from cutting transferred descriptors 4284 to 1650. **(c) Race-free async scheduling** (vLLM PRs #48481/#45357, `--async-scheduling`), credited as key but not isolated | ☐ SPIKE-first. **PIN MOVE REQUIRED, and it is the first deliverable:** every referenced vLLM PR post-dates our parity pin `555967922` (0.26.0.dev0), so none of this is mirrorable until the pin advances. The spike must (1) find the earliest upstream ref containing #40717 + #36687 + #48481, (2) prove that ref BUILDS AND RUNS a greedy golden on GB10 per [`oracle-gateability`](verification.md), (3) re-capture the SACRED goldens and show ZERO drift before any lever is ported. **Honest scoping:** upstream measured a GB200/NVLink72 DISAGGREGATED cluster (1xDEP8 decode, 4-8 prefill endpoints, 8192/1024, concurrency 64-5120) - one GB10 cannot reproduce it, so **25K is NOT adopted as our bar**. Only (a) is directly portable single-GPU; (c) overlaps the `VT_ASYNC_RUNNER` W3 work already default-OFF under C6; (b) is gated on the scale-out lane | | C11 | **Krea 2 image generation** (user-directed 2026-08-08, [krea-ai/krea-2](https://github.com/krea-ai/krea-2)): text to image diffusion transformer. Two community-licensed checkpoints, `krea/krea-2-raw` (base) and `krea/krea-2-turbo` (8-step distilled, up to 2k). Reference impl is PyTorch in-repo (`mmdit.py` MultiModalDiT + `autoencoder.py` + `sampling.py`) | ☐ SPIKE-first: arch inventory + reuse map against the MiniMax-H3 DiT lane, which already owns a joint video+audio diffusion transformer, an NVFP4/GGUF DiT weight-streaming loader and the `vllm_video_*` ABI slice - a text-to-image DiT is the SIMPLER sibling and should reuse the sampler / VAE-decode / DiT-forward seams rather than start new. **No pin move needed** (nothing upstream to track). **Upstream-mirror caveat:** vLLM has no Krea 2 path, so this is a from-scratch port owing a `porting-inventory.md` section 9 entry with the krea-2 repo as reference root and its PyTorch reference as the numerical oracle | +| C12 | **SGLang Breakable CUDA Graph (BCG), evaluated and mostly REFUTED** (user-directed 2026-08-17, from a public SGLang claim: *"first engine to develop and land BCG, the full CUDA Graph, and graph memory reuse"*). Grounded in the pinned SGLang tree `v0.5.15`/`f63458b5be` rather than in the claim text: BCG captures one forward as a SEQUENCE of `torch.cuda.CUDAGraph` segments split at eager break points on the attention and recurrent calls, with no `torch.compile` (`runner_backend_utils/breakable_cuda_graph/breakable_cuda_graph.py:204-241,244-260`; `runner_backend/breakable_cuda_graph_backend.py:14-17`). **The priority claim is narrower than it reads:** vLLM's v1 default is already `FULL_AND_PIECEWISE`, splitting at `splitting_ops` (`vllm/config/compilation.py:60-63,615,630` @ `555967922`), so the CAPABILITY is in our denominator; what SGLang landed first is the CONSTRUCTION — piecewise coverage with no Dynamo/Inductor/FX. | ☐ T1 (2026-08-17, [#1161](https://github.com/mudler/vllm.cpp/issues/1161)); spec [sglang-breakable-cuda-graph.md](specs/sglang-breakable-cuda-graph.md); row `SGLANG-BCG` in [sglang-matrix.md](sglang-matrix.md). **Four outcomes, three of them not throughput.** (a) *Dropping `torch.compile`* is NO lever here — we never had a compiler; `src/vt/cuda/cuda_backend.cu:203-232` is already raw stream capture. (b) *Prefill capture* is **REFUTED** on our CUDA gate models and recorded against `ENG-CUDAGRAPH` so nobody re-derives it: GB10 2026-07-09 measured prefill GPU-idle-between-launches at **3.8%** with GPU-busy >96% on both arms, and the 27B prefill gap at **92.5% non-GEMM glue GPU work** (dominant GEMM +0.17%, attention AHEAD). No launch bubbles to collapse. (c) *Graph memory reuse* is **real and unported** — `cudaGraphExecUpdate` appears nowhere in `src/` or `include/`, while we instantiate one exec per padded bucket across eight hand-rolled drivers → `ENG-CUDAGRAPH-DEDUP` `INVENTORIED` ([#1162](https://github.com/mudler/vllm.cpp/issues/1162)). (d) The analysis surfaced a STRUCTURAL row the claim did not: our capture is all-or-nothing (`runner.cpp:1338-1341` routes only `pure_decode`) and hand-rolled eight times → `ENG-CUDAGRAPH-BREAK` `INVENTORIED` ([#1163](https://github.com/mudler/vllm.cpp/issues/1163)), a COVERAGE row, not a throughput row. Diffusion looked like the one real lever (SGLang measured LTX-2 H200 10.75s→6.90s post-pin) but is **BLOCKED by our own defects**: an LTX-2.5 render does almost no device compute to capture — GPU util **exactly 0 in 321 of 347 samples** ([#1024](https://github.com/mudler/vllm.cpp/issues/1024)), no device arm for the VAE decode ([#1007](https://github.com/mudler/vllm.cpp/issues/1007)), and **57-66% of wall** in one resolution-CONSTANT serial host phase ([#1087](https://github.com/mudler/vllm.cpp/issues/1087)) → `ENG-CUDAGRAPH-DIFFUSION` `INVENTORIED` ([#1164](https://github.com/mudler/vllm.cpp/issues/1164)) | ## D. T2 (after T1, per porting-inventory.md) diff --git a/.agents/sglang-matrix.md b/.agents/sglang-matrix.md index 82259003c..917b51c4d 100644 --- a/.agents/sglang-matrix.md +++ b/.agents/sglang-matrix.md @@ -41,9 +41,9 @@ opt-in, separated from what is already covered and what is research/niche. |---|---| | FUSED | 23 | | SGLANG-DISTINCT | 8 | -| INVENTORIED | 5 | +| INVENTORIED | 6 | | OUT-OF-SCOPE | 8 | -| **Total** | **44** | +| **Total** | **45** | **Headline SGLANG-DISTINCT items (the opt-in worklist), ranked in the oracle spec §6:** cache-aware **LPM scheduling** (`--schedule-policy=lpm`, @@ -142,6 +142,12 @@ share the idea) or OUT-OF-SCOPE. | `SGLANG-REASONING` | reasoning parsers (deepseek-r1, harmony, think-tag) | `parser/reasoning_parser.py`, `parser/harmony_parser.py` | our 7 reasoning parsers (think_auto, deepseek_r1, mistral, minimax_m2, step3, olmo3), streamed as `reasoning` deltas | FUSED | Same reasoning-split-before-tool-parse model. | | `SGLANG-GRPC` | gRPC serving front-end (+ Anthropic / Ollama compat, Realtime/ASR) | `entrypoints/grpc_server.py:156`; `entrypoints/anthropic/`, `entrypoints/ollama/` | none (we ship the OpenAI HTTP + C-ABI surfaces) | OUT-OF-SCOPE | vLLM has no gRPC front-end either; niche protocol breadth, not a parity behavior. | +## Graph capture and runtime execution + +| ID | SGLang surface | SGLang anchor (`file:line`) | Our mapping / anchor | Class | Notes | +|---|---|---|---|---|---| +| `SGLANG-BCG` | **Breakable CUDA Graph** — one forward captured as a SEQUENCE of `torch.cuda.CUDAGraph` segments split at eager break points, with no `torch.compile`; plus per-phase backend selection (`FULL`/`BREAKABLE`/`TC_PIECEWISE`/`DISABLED`, decode defaults `FULL`, prefill defaults `BREAKABLE` on CUDA) and graph-executable dedup via `cudaGraphExecUpdate` | `runner_backend_utils/breakable_cuda_graph/breakable_cuda_graph.py:204-241` (segment split), `:244-260` (replay), `:14-23,156-169` (shared mempool + weak-ref intermediates); `runner_backend/breakable_cuda_graph_backend.py:14-17` ("No torch.compile"); break points `layers/radix_attention.py:256`, `attention_forward_methods/forward_mla.py:1092`, `layers/radix_linear_attention.py:159`, `models/nemotron_h.py:1240`, `dsa/dsa_indexer.py:2414`; config `cuda_graph_config.py:38-45,95-112`; dedup `runner_backend/cuda_graph_dedup_mixin.py:219-242,358` | our capture primitive `src/vt/cuda/cuda_backend.cu:203-232`; bucket set `include/vllm/model_executor/models/decode_graph_sizes.h:32-41`; pure-decode routing `src/vllm/v1/worker/gpu/runner.cpp:1338-1341`; eight hand-rolled drivers (`Qwen3_5DecodeGraph`, `Qwen3_5DenseDecodeGraph`, `Qwen3MoeDecodeGraph`, `Qwen3DenseDecodeGraph`, `DeepseekV2DecodeGraph`, `VoxtralDecodeGraph`, `deepseek_v4.cpp`, `laguna.cpp`) | INVENTORIED | **The CAPABILITY is vLLM-derived, not SGLang-distinct**: vLLM's v1 default is already `FULL_AND_PIECEWISE`, splitting at `splitting_ops` (`vllm/config/compilation.py:60-63,517,615,630` @ `555967922`), so it is in our benchmark denominator. What SGLang landed FIRST is the CONSTRUCTION — piecewise coverage from runtime stream capture plus a decorator, no Dynamo/Inductor/FX. That construction is the part worth mirroring, because we have no compiler either. **Prefill capture is REFUTED as a lever here** (GB10 2026-07-09: prefill GPU-idle-between-launches 3.8%, GPU-busy >96%, 27B gap 92.5% non-GEMM GLUE) — do not re-derive. Exec dedup is unported (`cudaGraphExecUpdate` appears nowhere in `src/` or `include/`). Spec [sglang-breakable-cuda-graph.md](specs/sglang-breakable-cuda-graph.md); issues [#1161](https://github.com/mudler/vllm.cpp/issues/1161), [#1162](https://github.com/mudler/vllm.cpp/issues/1162), [#1163](https://github.com/mudler/vllm.cpp/issues/1163), [#1164](https://github.com/mudler/vllm.cpp/issues/1164). | + ## Structurally distinctive / research (vLLM has no native analogue) | ID | SGLang surface | SGLang anchor (`file:line`) | Our mapping / anchor | Class | Notes | diff --git a/.agents/specs/sglang-breakable-cuda-graph.md b/.agents/specs/sglang-breakable-cuda-graph.md new file mode 100644 index 000000000..73bbf83b4 --- /dev/null +++ b/.agents/specs/sglang-breakable-cuda-graph.md @@ -0,0 +1,191 @@ +# SGLang Breakable CUDA Graph: what it is, and what it is worth here + +**Owning rows:** `SGLANG-BCG` in [sglang-matrix.md](../sglang-matrix.md) for the +inventory row; `ENG-CUDAGRAPH` in [engine-matrix.md](../engine-matrix.md) for the +prefill verdict this spec records against it. +**Issue:** [#1161](https://github.com/mudler/vllm.cpp/issues/1161). +**Recommendation:** record the surface, record the prefill refutation, and derive +three rows. One of the three is blocked on work that is already open. + +This spec exists because a public claim reached the operator and needed grounding +before it could change any plan. The claim: *"SGLang is the first engine to develop +and land Breakable CUDA Graph (BCG), the full CUDA Graph, and graph memory reuse. +BCG drops torch.compile for faster setup and broader compatibility. Full graph capture +brings prefill latency down on dynamic workloads. Memory reuse keeps the graph +footprint fixed as coverage grows."* + +Every claim below was read in the pinned tree. None was taken from the claim text. + +## Now + +`ANALYSIS`. No engine code is in scope here. The derived work is under +[`## Owed`](#owed) and each item has its own issue. + +## Pins + +| Component | Pin | Where | +|---|---|---| +| SGLang | tag `v0.5.15`, commit `f63458b5beaceabbd9d749b9fc956370e1b649e6` | [oracles/sglang.md](../oracles/sglang.md); paths below are relative to `python/sglang/srt/` | +| vLLM | `555967922` / 0.26.0.dev0 | [upstream-sync.md](../upstream-sync.md); the mirror source | +| Ours | `affc2a7fd` | the base of this branch | + +Two SGLang commits cited under [§5](#5-diffusion-is-the-one-shape-where-the-lever-is-real-and-it-is-blocked) +post-date the pin. They are cited as **dated upstream events**, never as pinned +evidence, and no gate here depends on them. + +## 1. What BCG is + +BCG captures one forward as a **sequence** of `torch.cuda.CUDAGraph` segments split at +eager break points, not as one graph. + +| Mechanism | Anchor | +|---|---| +| `eager_on_graph(True)` ends the current segment, runs the marked call eagerly once, records a `replay_fn` closure over weak-ref'd args, and begins a new segment | `runner_backend_utils/breakable_cuda_graph/breakable_cuda_graph.py:204-241` | +| Replay is `for seg: seg.replay(); break_fns[i]()` | `:244-260` | +| Every segment shares one mempool; intermediates are weak-ref'd, so the pool reclaims across segments and the activation footprint does not grow with segment count | `:14-23`, `:156-169` | +| "segment-captured graphs with eager break markers. No torch.compile." | `runner_backend/breakable_cuda_graph_backend.py:14-17` | + +The break points are the attention and recurrent calls: `layers/radix_attention.py:256`, +`models/deepseek_common/attention_forward_methods/forward_mla.py:1092`, +`layers/radix_linear_attention.py:159`, `models/nemotron_h.py:1240`, +`layers/attention/dsa/dsa_indexer.py:2414`. + +"Graph memory reuse" is **two** mechanisms, and the claim text merges them. One is the +shared mempool above. The other is executable dedup at +`runner_backend/cuda_graph_dedup_mixin.py:219-242`: hash each captured graph's +topology, and on a signature hit re-point one shared `cudaGraphExec` with +`cudaGraphExecUpdate` instead of instantiating a second one. It logs +`"captured %d CUDA graphs, deduped to %d execs"` at `:358`. + +Backend choice is per phase. `cuda_graph_config.py:38-45` offers +`FULL | BREAKABLE | TC_PIECEWISE | DISABLED`; decode defaults to `FULL`, prefill +defaults to `BREAKABLE` on CUDA (`:95-112`). + +## 2. The priority claim, stated accurately + +vLLM's v1 default is already `FULL_AND_PIECEWISE`: full graph for decode, piecewise +graph split at `splitting_ops` for prefill and mixed +(`vllm/config/compilation.py:60-63,517,615,630`). The **capability** has been in the +mirror source, and therefore in our benchmark denominator, for a long time. + +What SGLang landed first is the **mechanism**: piecewise coverage from runtime stream +capture plus a decorator, with no Dynamo, no Inductor and no FX splitting. That is a +real result and it removes a compiler dependency. It is not new engine behavior. A +reader of the claim text would conclude otherwise, which is why this section exists. + +## 3. Claim by claim, against our measurements + +| Claim | Verdict here | Basis | +|---|---|---| +| BCG drops `torch.compile` for faster setup and broader compatibility | **No lever.** We never had a compiler. `src/vt/cuda/cuda_backend.cu:203-232` is already raw `cudaStreamBeginCapture` / `cudaGraphInstantiate`. BCG is SGLang arriving where a C++ engine starts. | our own capture path | +| Full graph capture brings prefill latency down on dynamic workloads | **Refuted for our CUDA gate models.** | see below | +| Memory reuse keeps the graph footprint fixed as coverage grows | **Applies, and is unported.** | `grep -rn "cudaGraphExecUpdate" src include` is empty | + +### The prefill refutation + +Measured on GB10, 2026-07-09, both arms, same instrument, oracle graphed and +identity-asserted: + +- steady-state prefill **GPU-idle between launches is 3.8%**; GPU-busy is above 96% on + both arms. The one large raw idle gap is a single one-time JIT and capture window. +- the 27B prefill gap is **92.5% non-GEMM glue GPU work**. The dominant GEMM is the + same symbol at the same call count, +0.17%. Attention is where **we are ahead**. + +There are no launch bubbles in our prefill for a graph to collapse. vLLM's piecewise +prefill graph is already inside the denominator we lose to, so it does not explain the +gap either. Decode is a separate case and is already captured: batched decode at +concurrency 64 was about 24% launch overhead and is already one `cudaGraphLaunch` per +step. + +**This is a negative result and it is recorded so nobody re-derives it.** It does not +generalize to a path that is currently eager and currently host-bound. Any future +prefill-graph proposal must name such a path and measure it first. + +## 4. The structural finding, which is not a speed finding + +Our capture is all-or-nothing. `src/vllm/v1/worker/gpu/runner.cpp:1338-1341` routes +only `pure_decode` batches to a graph, so one host-dependent op forces the whole step +eager. Coverage is a cliff, not a slope, and +[#1020](https://github.com/mudler/vllm.cpp/issues/1020) is one instance of it. + +Eight call sites hand-roll capture: `Qwen3_5DecodeGraph`, `Qwen3_5DenseDecodeGraph`, +`Qwen3MoeDecodeGraph`, `Qwen3DenseDecodeGraph`, `DeepseekV2DecodeGraph`, +`VoxtralDecodeGraph`, plus graph code in `deepseek_v4.cpp` and `laguna.cpp`. Each +re-derives capture, bucket padding, persistent-input threading and the pure-decode +predicate. AGENTS.md names this shape: a parallel path written by hand instead of a +shared seam. + +BCG is the right construction for the fix, because vLLM gets its split from +`torch.compile` and we have no compiler. The value is **coverage and one seam instead +of eight**. It is not throughput on its own, and it must not be sold as throughput. + +## 5. Diffusion is the one shape where the lever is real, and it is blocked + +Our diffusion path captures no graphs at all: `grep` for capture across +`src/vllm/model_executor/models/ltx2*.cpp` returns nothing. A denoise loop is the ideal +capture target, and SGLang measured large wins on exactly it after our pin: LTX-2 +two-stage H200 e2e 10.75 s to 6.90 s (`d4be483efb`), SANA 1024px e2e -26% +(`6c7498113f`), SANA denoise 0.73 s to 0.457 s (`56ef810cad`). + +**Discount the magnitude hard**, because their win is mostly PyTorch host tax per +denoise step, which our C++ loop does not pay. + +**The premise fails anyway.** A denoise-loop graph needs a denoise loop that runs on the +device, and ours does not: + +- [#1024](https://github.com/mudler/vllm.cpp/issues/1024) measured GPU utilization + **exactly 0 in 321 of 347 samples** on a `--device cuda` render, with all 26 non-zero + samples inside the staging window, and the process holding exactly 1.00 core of 20 + for over 17 minutes after staging. +- [#1007](https://github.com/mudler/vllm.cpp/issues/1007): the video VAE decode has no + device arm. +- [#1087](https://github.com/mudler/vllm.cpp/issues/1087): **57-66% of render wall** is + one single-threaded phase measuring 1731 s and 1732 s across two rungs whose voxel + counts differ 2.75x. [#1010](https://github.com/mudler/vllm.cpp/issues/1010) is open + because nothing timestamps a phase boundary, so the phase is not yet named. + +A graph collapses host launch dispatch. There is close to no device dispatch here to +collapse. **Capturing now would measure nothing.** + +## 6. What was deliberately not done + +- No engine code, no capture change, no measurement taken. +- No claim that BCG makes us faster. Three of the four candidate levers are refuted, + structural, or blocked. +- No oracle run. SGLang remains `gateable = no` on this project's hardware for + correctness; the perf oracle stood up separately under `SGLANG-ORACLE-PERF`. +- No advance of the SGLang pin. The two post-pin diffusion commits are dated events, + not evidence. + +## Owed + +| Item | Issue | Row | +|---|---|---| +| Graph executable dedup via `cudaGraphExecUpdate` | [#1162](https://github.com/mudler/vllm.cpp/issues/1162) | `ENG-CUDAGRAPH-DEDUP` | +| A shared capture seam with break points, retiring the eight hand-rolled drivers | [#1163](https://github.com/mudler/vllm.cpp/issues/1163) | `ENG-CUDAGRAPH-BREAK` | +| Denoise-loop capture, blocked on #1010, #1087, #1024, #1007 | [#1164](https://github.com/mudler/vllm.cpp/issues/1164) | `ENG-CUDAGRAPH-DIFFUSION` | + +## Stop conditions + +- Stop if a measurement contradicts §3. The refutation is dated and hardware-specific, + and a negative result is regime-dependent. +- Stop `ENG-CUDAGRAPH-DIFFUSION` at step 4 of its issue if the denoise loop turns out + GPU-bound once device-resident. Close it the way this spec closed prefill. +- Stop any row that starts claiming throughput from §4 without naming a measured + host-bound path first. + +## Outcome + +Recorded 2026-08-17. The claim is accurate about the mechanism and overstated about +priority. Its three parts land here as one no-lever, one refutation, and one unported +memory optimization, plus one structural row and one blocked row that the analysis +surfaced rather than the claim. + +What was rejected and why: porting BCG as a prefill throughput lever, because our +prefill has 3.8% host idle and is 92.5% glue-GPU-bound; and scoping the diffusion +capture now, because the render does no device compute to capture. + +Why the defaults are what they are: `ENG-CUDAGRAPH` stays `PARTIAL` rather than moving, +because nothing shipped. `SGLANG-BCG` is classified `INVENTORIED` rather than `OWED`, +because the capability it names is already in the mirror source and only its +construction is distinct. diff --git a/scripts/check-agent-record.py b/scripts/check-agent-record.py index 5ea2ce53b..cfcff3fe2 100644 --- a/scripts/check-agent-record.py +++ b/scripts/check-agent-record.py @@ -518,7 +518,7 @@ # Issue #632; `SPIKE` on its committed spec. The row claims no implementation: no # parser, no baseline and no test exists yet. # Bumped for a real new row, never to make a failing state transition pass. -ENGINE_ROWS = 157 +ENGINE_ROWS = 160 ENGINE_SUMMARY_SECTIONS = ( ("Engine and scheduling", "Engine core and scheduling"),