diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md index b8fd212ea..f690aaf52 100644 --- a/.agents/benchmark-record.md +++ b/.agents/benchmark-record.md @@ -25988,3 +25988,52 @@ matched the GEMMs' measured 3.98 TFLOP/s, its 0.140 TFLOP would cost **35.2 ms instead of 688.1 ms**: forward 1569 -> 916 ms (**1.71x on the DiT**), bucket 370.510 -> 216.3 s, run 598.207 -> 444.0 s (**1.35x end to end**). An upper bound from a flop ratio on a kernel nobody has written. +### The Qwen3.5-4B `1.0283x` row ran against an UNFUSED denominator (#414, #1345) + +Found 2026-08-19 while landing [#607](https://github.com/mudler/vllm.cpp/issues/607) +wave L4, which gated the CLI oracle-launch surface. This leg is on the OTHER +surface, so the new checker does not see it, and it is the one published figure +the L4 sweep found still carrying the #414 defect unmarked. + +The chain, each link checked rather than assumed: + +1. `docs/BENCHMARKS.md` cites `docs/bench-evidence/qwen35-4b-sm120-main-20260807.md`, + whose reproduction identity names `Qwen/Qwen3.5-4B` and the exact workload + (128 requests, 128 output tokens, concurrency 32, + `max_num_batched_tokens=2048`, greedy) that `tools/bench/run_qwen35_4b_compare.sh` + drives. +2. `tools/bench/run_qwen35_4b_compare.sh:120` runs the vLLM arm through + `tools/bench/vllm_closed_loop_metrics.py`. Its `LLM(...)` at `:160` never + passes `language_model_only` and exposes no argument that could set it. +3. Upstream registers `Qwen/Qwen3.5-4B` under `Qwen3_5ForConditionalGeneration` + (`tests/models/registry.py:1322-1324`, `extras={"4b": "Qwen/Qwen3.5-4B"}`), so + `multimodal_config` is non-`None` and `text_only` resolves `False` + (`vllm/config/multimodal.py:78`, `vllm/model_executor/models/qwen3_next.py:325`). +4. The conjunct was live at the time of the run. `git log -S'use_fused_qk_norm_rope_gate'` + over `vllm/model_executor/models/qwen3_next.py` dates its introduction to + `16282a9c4` on 2026-06-10, two months before the 2026-08-07 measurement. +5. The remaining conjuncts hold: sm_120 is CUDA, and Qwen3.5-4B carries + `attn_output_gate` with NeoX-style RoPE. + +So the oracle issued four ops per full-attention layer where its own production +configuration issues one, against our single fused launch. + +**Direction: it flatters us**, for the reason #414 gives. **Magnitude: NOT +MEASURED, and deliberately not estimated.** The model is GDN-hybrid, so only its +full-attention layers are exposed, and the L4 wave took no measurement: both +fleet devices were held and it carried no lease authority. Quoting a corrected +ratio here would be a number nobody took. + +The row keeps its values and gains the attribution, per AGENTS.md: evidence is +annotated, never deleted. Re-measurement is owed with +[#1345](https://github.com/mudler/vllm.cpp/issues/1345), which also owns the +repair to the three in-process harnesses. + +**Not affected, so the next reader does not re-derive it.** #414 reaches a figure +only where the full-attention layers are `Qwen3NextAttention` AND the checkpoint +loads as a `*ForConditionalGeneration`. That is the Qwen3.5/3.6/3.8 family alone. +The OPT, GLM-4, InternLM2, Qwen3-dense, Qwen3-Coder and DeepSeek-V2-Lite legs +never construct `Qwen3NextAttention`. The `Qwen3.8-27B` rows ran through +`tools/bench/run_serve_low.py`, which has passed `--language-model-only` since it +was written. The Qwen3.5-4B GDN prefill kernel row is conv and post-conv timing +on the LINEAR-attention path, which the full-attention preamble does not touch. diff --git a/.agents/engine-matrix.md b/.agents/engine-matrix.md index 3463e7146..d3bd8ba65 100644 --- a/.agents/engine-matrix.md +++ b/.agents/engine-matrix.md @@ -74,7 +74,7 @@ forensics: roadmap_v1.md and the parity ledger. | `ENG-DBO-UBATCH` | DBO and ubatch overlap | T2 | `vllm/config/parallel.py:208,524` | - | - | `planned: specs/dbo-ubatch.md` | `INVENTORIED` | - | | `ENG-MOE-SHARED-AUX` | MoE shared-expert MLP on an aux CUDA stream concurrent with the routed-expert router/align/grouped-GEMMs (mirror vLLM's decode overlap; the largest remaining 35B c1/c2 engine lever). Fork the shared MLP onto a 2nd persistent per-device stream, join before the combine → byte-identical to serial (independent shared/routed paths both complete before combine; overlap changes WHEN not WHAT). Gated `T <= threshold` decode + CUDA. The aux stream draws scratch from a SEPARATE `AuxPool` so the concurrent main-stream routed allocations never share a live block with it (the `DevicePool` reuse invariant is single-stream ordering; vLLM sidesteps this with its stream-aware caching allocator's `record_stream`). `VT_MOE_SHARED_AUX_STREAM` **DEFAULT ON** (`=0` rollback); `VT_MOE_SHARED_AUX_THRESHOLD` (default 128; GB10 48-SM calibration). Captured in the decode CUDA-graph via the fork/join event edges (`ThreadLocal` capture, no abort). Only the committed Marlin MoE decode path; wmma fallback/CPU/GGUF and 27B dense unaffected | T1 | `vllm/model_executor/layers/fused_moe/runner/shared_experts.py:99-104,125-142`; `vllm/utils/multi_stream_utils.py:20-58` (`maybe_execute_in_parallel`, TRT-LLM port); `vllm/utils/torch_utils.py:736-756` (`aux_stream`); `vllm/envs.py:260` (threshold 256) | fork/join `src/vllm/model_executor/models/qwen3_5.cpp:3999,4114` (`MoeBlockFusedMarlinCuda`); aux stream+events `src/vllm/model_executor/models/qwen3_5.cpp:3575,3581` (`MoeAuxStream`/`MoeAuxStreamFor`); predicates `:3553,3560`; aux-pool isolation `:496,3538` (`AuxPool`/`ActivePool`/`ActivePoolScope`) + `DBuf pool_` routing `:645` | **DGX (prod flags, one flock):** overlap ON==OFF BYTE-IDENTICAL — `tests/parity/test_qwen36_paged_engine.cpp:116` 35B **315/315** + `tests/parity/test_qwen27_paged_engine.cpp` 27B **235/235** under `VT_MOE_SHARED_AUX_STREAM`∈{0,1}; captured-vs-eager (`VLLM_CPP_CUDAGRAPH=0`, ON) 315/315; shipping default (no env) 315/315+235/235, rollback `=0` 315/315+235/235; `compute-sanitizer memcheck` (default ON, captured) 0 errors; in-situ interleaved TPOT A/B (drop cold rep1) c1 −5.6% / c2 −2.7% / c4 −3.7% / c8 −3.4% / c16 −1.6% / c32 −1.5% (WINS every conc, zero regression); ledger [parity-ledger.md](parity-ledger.md) 2026-07-19 row | [moe-shared-aux-stream.md](specs/moe-shared-aux-stream.md) | `ANCHOR-BACKFILL` | `CLAIM-MOE-SHARED-AUX-1` | | `ENG-RUNNER-MODELSHAPE` | **Runner is model-shape-agnostic over the KV-cache group structure** — the extensibility deliverable the first additive-model bring-up (Qwen3 dense) forced. Before W1 the `GPUModelRunner` had only ever executed the Qwen3.6 HYBRID topology and hardcoded it in two places: (#1) the KV-buffer alloc loop indexed `config_.layer_types[l]`, out-of-bounds on a pure-dense model's EMPTY `layer_types`; (#2) each `execute_model` step unconditionally built the GDN metadata (`gather_block_table(gdn_group_id_)` / `remap_gdn_state_slots` / `GDNAttentionMetadataBuilder`), which reads `block_table[-1]` when there is no mamba group. W1 drives both off the resolved KV-group structure — a model-agnostic `has_mamba_group` / `gdn_group_id_ >= 0` predicate (NOT a model-name check): empty/absent `layer_types` ⇒ all full-attention; no mamba group ⇒ the whole GDN metadata/state path is skipped and `gdn_meta` stays default-empty. A full-attention-only KV config (one FA group, no MambaSpec) now allocates + steps cleanly; the hybrid gate models keep their GDN group so their path is BYTE-IDENTICAL. This is a one-time generalization: every future dense/non-hybrid arch (Llama, Mistral) now adds new-files-only, zero further runner edits. **PER-LAYER KV head_dim extension (Gemma-4 G1b, 2026-07-28, `CLAIM-GEMMA4-G1B`):** the runner's full-attn alloc/view loops now consume an OPTIONAL `KVCacheConfig::per_layer_attn_specs` (index == layer) so a HETEROGENEOUS-head_dim model (Gemma-4: sliding 256 / global 512, same num_kv_heads) sizes each non-GDN layer's paged KV + PagedKvCache view from its OWN spec. The field is EMPTY for every uniform-KV model ⇒ the loop collapses to the single group spec ⇒ byte-identical allocation/view/indexing/dispatch (same additive-identical property as the model-shape generalization above). Block table / KV manager / scheduler stay head_dim-independent (num_blocks + block_size, uniform) so no per-group block table is introduced | T0 | model-agnostic runner drives off `kv_cache_config.kv_cache_groups` — `vllm/v1/worker/gpu/model_runner.py` `initialize_kv_cache` / attention-metadata build (per-group, no hardcoded hybrid) @ `e24d1b24` | `src/vllm/v1/worker/gpu/runner.cpp:458-470` (alloc loop: `has_mamba_group && !layer_types.empty()` gate) + `:651-680` (GDN metadata build gated on `gdn_group_id_ >= 0`, default-empty `gdn_meta` otherwise); per-layer KV head_dim: `include/vllm/v1/kv_cache_interface.h` (`KVCacheConfig::per_layer_attn_specs`) consumed in `src/vllm/v1/worker/gpu/runner.cpp` `initialize_kv_cache` (per-layer `FaDims` alloc+view), published by `src/vllm/model_executor/models/gemma4_registry.cpp` (`MakeGemma4ForConditionalGenerationKVCache`); the full-attention-only KV spec that exercises the base path `src/vllm/model_executor/models/qwen3_dense.cpp` (`MakeQwen3ForCausalLMKVCache`) | `tests/vllm/v1/worker/test_runner.cpp:1129` — "full-attention-only KV config allocates without the GDN path" + "full-attention-only step skips GDN metadata build (no OOB)" (RED→GREEN: both SIGSEGV pre-generalization; GREEN post). Behaviour-preservation gate: DGX **27B 235/235 + 35B 315/315 UNCHANGED** under the fix; per-layer-KV inertness: full CPU runner/KV suite green + **OLMo-2 SACRED GPU re-gate 16/16 UNCHANGED**; heterogeneous path proven by **Gemma-4 E4B STRICT 32/32** (`tests/parity/test_gemma4_paged_engine.cpp`); ASan/UBSan clean on the affected paths | [first-additive-model-qwen3-dense.md](specs/first-additive-model-qwen3-dense.md) §3 (seam gaps #1/#2), §6 (W1); [gemma4-multimodal.md](specs/gemma4-multimodal.md) §G1b | `ACTIVE` | `CLAIM-MODEL-QWEN3-DENSE` | -| `ENG-MM-INPUT-PIPELINE` | **Multimodal INPUT pipeline + encoder-cache engine seam (M1), INERT when no mm input.** The C++ mirror of `vllm/multimodal/`: `MultiModalKwargs`/`MultiModalFeatureSpec`/`MultiModalInputs`, the `MultiModalHasher` mm-hash (blake3), the Qwen3-VL image processor (smart_resize + fused rescale/normalize + patchify -> `pixel_values`+`image_grid_thw`) and placeholder-token expansion, plus the `EncoderCacheManager` (+`ComputeMmEncoderBudget`) and the LMCache `extra_keys` seam. Additive `mm_features` carried on `Request`/`EngineCoreRequest`; with NO mm input every field is empty and every path is byte-identical to the text engine. Processor output is BIT/BYTE-identical to the vLLM 0.25.0 oracle (M0 fixture). Does NOT build the vision tower / embed-merge (M2). **SERVING wiring (ROAD-V1-MM `MM-SERVE-ENGINE`, 2026-07-28, `CLAIM-MM-SERVING-W2`):** the OpenAI server now carries the parsed `MultiModalInputs` into the engine — additive `LLMEngine`/`AsyncLLM` `add_request(MultiModalInputs)`+`generate(MultiModalInputs)` overloads via `InputProcessor::process_inputs_mm` (mirror `input_processor.py:333-379`, empty mm_features == the tokens path), the chat-template placeholder-STRING helpers (`get_placeholder_str`/`_add_placeholder` mirror), and the serving_chat `MultiModalChatFn` seam (default unset ⇒ text byte-identical). **SEAM BODY (ROAD-V1-MM `MM-SERVE-E2E` W3, 2026-07-28, `CLAIM-MM-SERVING-E2E`):** `MakeQwen3VLImageChatFn` (chat_mm.cpp) is the seam body the server sets — messages → marker-inject → chat template → `EncodeWithSpecialTokens` (the single image_pad marker → one image_token_id) → `RouteImageRgb` EXPAND to 196 image tokens + mm_features; wired in `examples/server/main.cpp` (guarded on `preprocessor_config.json`; text-only unset ⇒ byte-identical). Gated `test_chat_mm` 8/8 + `test_openai_serving` (seam invoked + routed). **ENGINE MM-FORWARD LANDED (ROAD-V1-MM `MM-SERVE-E2E`, 2026-07-28, `CLAIM-ENGINE-MM-FORWARD`):** the engine model runner now HAS an mm forward — `ModelForwardInput` gains an ADDITIVE default-nullopt `std::optional mm` (merged inputs_embeds + 3-D MRoPE positions + DeepStack, borrowed handles; nullopt-for-text ⇒ shared runner path byte-identical BY CONSTRUCTION), `Qwen3VLForConditionalGeneration` is `REGISTER_VLLM_MODEL`-registered (`qwen3_vl_registry.cpp`), and the registered forward FOLDS the M2c decode into `ModelRegistry::Forward` via the SHARED `Qwen3VLForwardStepLastLogits` (`Qwen3VLGenerateGreedyViaRegistry` drives every step through the registry). GPU token-exact gate `test_qwen3vl_registry_e2e` (image→text THROUGH `ModelRegistry::Forward` == M2c golden 32/32 STRICT, dgx.casa GB10); text inertness `test_runner` 16/16 + `test_scheduler` 36/36 + `test_model_registry` 24/24 + `test_chat_mm` 8/8 + `test_openai_serving` 41/41 all green. RESIDUAL: the FULL in-runner scheduler-fed tower run (batched-loop mm building the field from staged encoder outputs) + the real server `/v1/chat/completions` GPU e2e — recipe in `specs/mm-serving.md`. **INPUT LIMITS L1 LANDED (#607, 2026-08-13):** the per-modality `limit_per_prompt` + `GetLimitPerPrompt` precedence (`language_model_only` ⇒ 0 BEFORE the map, else the map, else 999) and the refusal that gives those numbers effect — `AllowedMmLimits` folding by `min()` against the model's own ceiling, `ValidateNumItems` with upstream's exact message, and both call sites with the `enable_mm_embeds` escape. NO serve surface and NO live call site: nothing constructs a `MultiModalConfig` on a request yet, which is L2's. **INPUT LIMITS L2 LANDED (#607, #686, 2026-08-14):** the flags (`--[no-]language-model-only`, `--limit-mm-per-prompt ''`; `arg_utils.py:555-556,1276-1279,1691-1692` over `ParseLimitMmPerPromptJson`, the port of `multimodal.py:212-236` + the DummyOptions dataclasses `:17-43`), the C-ABI fields (`vllm_model_params.language_model_only`/`.limit_mm_per_prompt`, **ABI v19**), and the LIVE CALL SITE: `ValidateChatMmLimits` (`chat_utils.py:648-662`) runs as step 0 of `MakeQwen3VLImageChatFn` over a `BaseProcessingInfo` folding `LoadedEngine::mm_config()` with `Qwen3VLChatSupportedMmLimits() == {"image": 1}` — the seam's own ceiling, which is the `min()` fold operand #686 recorded as undeclared. A three-image request is now HTTP 400 with upstream's message rather than an opaque 500 / a truncated answer. NOT claimed: any memory win (L3 gates tower construction, unmeasured). Still unwired: the `process_inputs_mm` call site (`context.py:461`), blocked on the per-model `get_supported_mm_limits()` hook. | T1 | `vllm/multimodal/{inputs.py,hasher.py:50,processing/processor.py:1663,processing/inputs.py:62}`; `vllm/model_executor/models/qwen3_vl.py:{1400,1233}`; `vllm/v1/core/encoder_cache_manager.py:17`; transformers `image_processing_qwen2_vl.py:62`, `image_processing_backends.py:327`; tests `tests/multimodal/test_processing.py`, `tests/multimodal/test_hasher.py`, `tests/v1/core/test_encoder_cache_manager.py` @ `e24d1b24` | `src/vllm/multimodal/hasher.cpp`, `src/vllm/multimodal/qwen3vl_processor.cpp`, `include/vllm/multimodal/{inputs.h,hasher.h,qwen3vl_processor.h}`; `src/vllm/v1/core/encoder_cache_manager.cpp` + `include/vllm/v1/core/encoder_cache_manager.h`; additive inert fields `include/vllm/v1/request.h` + `src/vllm/v1/request.cpp` + `include/vllm/v1/engine/types.h`; `extra_keys` seam `include/vllm/v1/kv_offload/lmcache/chunked_token_database.h` + `.cpp`; M0 `scripts/mm/m0_oracle_capture.py`; L1 limits `include/vllm/config/multimodal.h` + `include/vllm/multimodal/processing/context.h` + `src/vllm/multimodal/processing/context.cpp`, refusal type relocated to `include/vllm/v1/engine/validation_error.h`; L2 flags+ABI+call site `src/vllm/config/multimodal.cpp` (`ParseLimitMmPerPromptJson`) + `src/vllm/entrypoints/openai/server_main.cpp` + `include/vllm.h` (ABI v19) + `src/capi/vllm_c.cpp` + `EngineParams::multimodal`/`LoadedEngine::mm_config()` + `src/vllm/entrypoints/openai/chat_mm.cpp` (`ChatPartModality`, `ValidateChatMmLimits`, `Qwen3VLChatSupportedMmLimits`) — anchors `src/vllm/multimodal/hasher.cpp:56`, `src/vllm/config/multimodal.cpp:49`, `src/vllm/entrypoints/openai/chat_mm.cpp:295,311`, `include/vllm.h:197,403` | `tests/vllm/multimodal/test_qwen3vl_processor.cpp` (processor-parity 23/23 BIT-identical vs the M0 oracle fixture `tests/vllm/multimodal/fixtures/qwen3vl/`, RED-first: wrong normalize shift -> 1.2M mismatches); `tests/vllm/v1/core/test_encoder_cache_manager.cpp` 32/32. Text-inertness: `test_request`/`test_engine_types`/`test_lmcache_codec`/`test_lmcache_key_agreement`/`test_openai_conformance` all green standalone; SACRED CUDA 27B/35B/Coder = GPU inertness proof; `check-device-leakage` OK — anchor `tests/vllm/multimodal/test_qwen3vl_processor.cpp:59`. L1 limits: `tests/vllm/config/test_multimodal_config.cpp` 7/7 (21 assertions) + `tests/vllm/multimodal/test_processing_limits.cpp` 19/19 (78 assertions), porting `tests/multimodal/test_processing.py:902-941,944-985`, `tests/entrypoints/multimodal/llm/test_mm_embeds_only.py:41-49` and `tests/entrypoints/unit_tests/test_chat_utils.py:1498-1560` @ `5559679229bc`; mutations proven RED: map-before-flag precedence, the dropped throw, the dropped `min()` fold. L2 (aarch64 `build-test-cpu-arm64` lane, `-DVLLM_CPP_CUDA=OFF`): `tests/vllm/entrypoints/openai/test_serve_mm_limits.cpp` 11/11 (109 assertions, flags + the parser's upstream refusals + the builtin-only reach of `extra="forbid"`) + `test_chat_mm` 11/11 (126) + `test_openai_api_server` 56/56 (CASES; its assertion count is timing-dependent — 632/648/651 across three runs of one binary, so only the case count is quotable — the HTTP 400 arm proven against BOTH a 500 and a truncated 200); RED-first behavioural: `CHECK(500 == 400)`, `CHECK("InternalServerError" == BadRequestError)` and `CHECK(200 == 400)`; mutations proven RED: flag→config plumbing dropped, the call-site wiring dropped, and the refusal re-typed off `InputValidationError` (which lands as the 500 L1's design avoided) | [multimodal-track.md](specs/multimodal-track.md) §3 (M0/M1) | `READY` | - | +| `ENG-MM-INPUT-PIPELINE` | **Multimodal INPUT pipeline + encoder-cache engine seam (M1), INERT when no mm input.** The C++ mirror of `vllm/multimodal/`: `MultiModalKwargs`/`MultiModalFeatureSpec`/`MultiModalInputs`, the `MultiModalHasher` mm-hash (blake3), the Qwen3-VL image processor (smart_resize + fused rescale/normalize + patchify -> `pixel_values`+`image_grid_thw`) and placeholder-token expansion, plus the `EncoderCacheManager` (+`ComputeMmEncoderBudget`) and the LMCache `extra_keys` seam. Additive `mm_features` carried on `Request`/`EngineCoreRequest`; with NO mm input every field is empty and every path is byte-identical to the text engine. Processor output is BIT/BYTE-identical to the vLLM 0.25.0 oracle (M0 fixture). Does NOT build the vision tower / embed-merge (M2). **SERVING wiring (ROAD-V1-MM `MM-SERVE-ENGINE`, 2026-07-28, `CLAIM-MM-SERVING-W2`):** the OpenAI server now carries the parsed `MultiModalInputs` into the engine — additive `LLMEngine`/`AsyncLLM` `add_request(MultiModalInputs)`+`generate(MultiModalInputs)` overloads via `InputProcessor::process_inputs_mm` (mirror `input_processor.py:333-379`, empty mm_features == the tokens path), the chat-template placeholder-STRING helpers (`get_placeholder_str`/`_add_placeholder` mirror), and the serving_chat `MultiModalChatFn` seam (default unset ⇒ text byte-identical). **SEAM BODY (ROAD-V1-MM `MM-SERVE-E2E` W3, 2026-07-28, `CLAIM-MM-SERVING-E2E`):** `MakeQwen3VLImageChatFn` (chat_mm.cpp) is the seam body the server sets — messages → marker-inject → chat template → `EncodeWithSpecialTokens` (the single image_pad marker → one image_token_id) → `RouteImageRgb` EXPAND to 196 image tokens + mm_features; wired in `examples/server/main.cpp` (guarded on `preprocessor_config.json`; text-only unset ⇒ byte-identical). Gated `test_chat_mm` 8/8 + `test_openai_serving` (seam invoked + routed). **ENGINE MM-FORWARD LANDED (ROAD-V1-MM `MM-SERVE-E2E`, 2026-07-28, `CLAIM-ENGINE-MM-FORWARD`):** the engine model runner now HAS an mm forward — `ModelForwardInput` gains an ADDITIVE default-nullopt `std::optional mm` (merged inputs_embeds + 3-D MRoPE positions + DeepStack, borrowed handles; nullopt-for-text ⇒ shared runner path byte-identical BY CONSTRUCTION), `Qwen3VLForConditionalGeneration` is `REGISTER_VLLM_MODEL`-registered (`qwen3_vl_registry.cpp`), and the registered forward FOLDS the M2c decode into `ModelRegistry::Forward` via the SHARED `Qwen3VLForwardStepLastLogits` (`Qwen3VLGenerateGreedyViaRegistry` drives every step through the registry). GPU token-exact gate `test_qwen3vl_registry_e2e` (image→text THROUGH `ModelRegistry::Forward` == M2c golden 32/32 STRICT, dgx.casa GB10); text inertness `test_runner` 16/16 + `test_scheduler` 36/36 + `test_model_registry` 24/24 + `test_chat_mm` 8/8 + `test_openai_serving` 41/41 all green. RESIDUAL: the FULL in-runner scheduler-fed tower run (batched-loop mm building the field from staged encoder outputs) + the real server `/v1/chat/completions` GPU e2e — recipe in `specs/mm-serving.md`. **INPUT LIMITS L1 LANDED (#607, 2026-08-13):** the per-modality `limit_per_prompt` + `GetLimitPerPrompt` precedence (`language_model_only` ⇒ 0 BEFORE the map, else the map, else 999) and the refusal that gives those numbers effect — `AllowedMmLimits` folding by `min()` against the model's own ceiling, `ValidateNumItems` with upstream's exact message, and both call sites with the `enable_mm_embeds` escape. NO serve surface and NO live call site: nothing constructs a `MultiModalConfig` on a request yet, which is L2's. **INPUT LIMITS L2 LANDED (#607, #686, 2026-08-14):** the flags (`--[no-]language-model-only`, `--limit-mm-per-prompt ''`; `arg_utils.py:555-556,1276-1279,1691-1692` over `ParseLimitMmPerPromptJson`, the port of `multimodal.py:212-236` + the DummyOptions dataclasses `:17-43`), the C-ABI fields (`vllm_model_params.language_model_only`/`.limit_mm_per_prompt`, **ABI v19**), and the LIVE CALL SITE: `ValidateChatMmLimits` (`chat_utils.py:648-662`) runs as step 0 of `MakeQwen3VLImageChatFn` over a `BaseProcessingInfo` folding `LoadedEngine::mm_config()` with `Qwen3VLChatSupportedMmLimits() == {"image": 1}` — the seam's own ceiling, which is the `min()` fold operand #686 recorded as undeclared. A three-image request is now HTTP 400 with upstream's message rather than an opaque 500 / a truncated answer. NOT claimed: any memory win (L3 gates tower construction, unmeasured). Still unwired: the `process_inputs_mm` call site (`context.py:461`), blocked on the per-model `get_supported_mm_limits()` hook. **KERNEL GATE L4 RESOLVED 2026-08-19 as a TRACKED EXCEPTION (#607, #414):** we do NOT mirror the `text_only` conjunct of `qwen3_next.py:324-331`, because mirroring is not representable at our seams. Upstream conjoins it because its fused Triton kernel indexes `cos_sin_cache` by 1-D `positions` and cannot express MRoPE (`qwen3_next.py:323`, `# TODO: support MRoPE`), falling back to an eager arm whose `self.rotary_emb` IS the MRoPE module. Ours are not those two arms: `vt::AttnQkNormRopeGate` takes NO positions, only a precomputed per-token `cos_sin` cache that `qwen3_5.cpp::BuildMropeCosSinHost` fills with the interleaved 3-section MRoPE selection, so our FUSED arm is the MRoPE arm while our eager arm (`vt::RopeNeox` on 1-D positions) has no MRoPE spelling. Conjoining `text_only` would select 1-D RoPE on exactly the configuration the conjunct protects and would break the landed M3-b image and M3d video STRICT 32/32 gates. Argued in `specs/multimodal-track.md` §1.6. What the exception does NOT excuse is the DENOMINATOR: #414's defect is a benchmark configuration, and `scripts/dgx-online-serving.sh` still launched the oracle without `--language-model-only` while `tools/bench/run_serve_low.py` passed it, so the two harnesses disagreed about the oracle's own configuration and the next canonical campaign would have reproduced the flattered ratios the 2026-08-13 series superseded. Both now pass it and `scripts/check-oracle-denominator-flags.py` keeps them agreeing. NO product code path changed, so no token gate and no measurement is claimed and no published number is withdrawn. Owed and filed in flow: #1340 (`VT_FUSE_ATTN_PREAMBLE=0` on the MRoPE path substitutes 1-D RoPE instead of refusing; needs a GPU VL token gate) and #1345 (the three in-process `LLM(...)` bench harnesses leave `language_model_only` at False with no way to set it). | T1 | `vllm/multimodal/{inputs.py,hasher.py:50,processing/processor.py:1663,processing/inputs.py:62}`; `vllm/model_executor/models/qwen3_vl.py:{1400,1233}`; `vllm/v1/core/encoder_cache_manager.py:17`; transformers `image_processing_qwen2_vl.py:62`, `image_processing_backends.py:327`; tests `tests/multimodal/test_processing.py`, `tests/multimodal/test_hasher.py`, `tests/v1/core/test_encoder_cache_manager.py` @ `e24d1b24` | `src/vllm/multimodal/hasher.cpp`, `src/vllm/multimodal/qwen3vl_processor.cpp`, `include/vllm/multimodal/{inputs.h,hasher.h,qwen3vl_processor.h}`; `src/vllm/v1/core/encoder_cache_manager.cpp` + `include/vllm/v1/core/encoder_cache_manager.h`; additive inert fields `include/vllm/v1/request.h` + `src/vllm/v1/request.cpp` + `include/vllm/v1/engine/types.h`; `extra_keys` seam `include/vllm/v1/kv_offload/lmcache/chunked_token_database.h` + `.cpp`; M0 `scripts/mm/m0_oracle_capture.py`; L1 limits `include/vllm/config/multimodal.h` + `include/vllm/multimodal/processing/context.h` + `src/vllm/multimodal/processing/context.cpp`, refusal type relocated to `include/vllm/v1/engine/validation_error.h`; L2 flags+ABI+call site `src/vllm/config/multimodal.cpp` (`ParseLimitMmPerPromptJson`) + `src/vllm/entrypoints/openai/server_main.cpp` + `include/vllm.h` (ABI v19) + `src/capi/vllm_c.cpp` + `EngineParams::multimodal`/`LoadedEngine::mm_config()` + `src/vllm/entrypoints/openai/chat_mm.cpp` (`ChatPartModality`, `ValidateChatMmLimits`, `Qwen3VLChatSupportedMmLimits`) — anchors `src/vllm/multimodal/hasher.cpp:56`, `src/vllm/config/multimodal.cpp:49`, `src/vllm/entrypoints/openai/chat_mm.cpp:295,311`, `include/vllm.h:197,403`; L4 denominator gate `scripts/check-oracle-denominator-flags.py` + the `--language-model-only` oracle arm of `scripts/dgx-online-serving.sh` | `tests/vllm/multimodal/test_qwen3vl_processor.cpp` (processor-parity 23/23 BIT-identical vs the M0 oracle fixture `tests/vllm/multimodal/fixtures/qwen3vl/`, RED-first: wrong normalize shift -> 1.2M mismatches); `tests/vllm/v1/core/test_encoder_cache_manager.cpp` 32/32. Text-inertness: `test_request`/`test_engine_types`/`test_lmcache_codec`/`test_lmcache_key_agreement`/`test_openai_conformance` all green standalone; SACRED CUDA 27B/35B/Coder = GPU inertness proof; `check-device-leakage` OK — anchor `tests/vllm/multimodal/test_qwen3vl_processor.cpp:59`. L1 limits: `tests/vllm/config/test_multimodal_config.cpp` 7/7 (21 assertions) + `tests/vllm/multimodal/test_processing_limits.cpp` 19/19 (78 assertions), porting `tests/multimodal/test_processing.py:902-941,944-985`, `tests/entrypoints/multimodal/llm/test_mm_embeds_only.py:41-49` and `tests/entrypoints/unit_tests/test_chat_utils.py:1498-1560` @ `5559679229bc`; mutations proven RED: map-before-flag precedence, the dropped throw, the dropped `min()` fold. L2 (aarch64 `build-test-cpu-arm64` lane, `-DVLLM_CPP_CUDA=OFF`): `tests/vllm/entrypoints/openai/test_serve_mm_limits.cpp` 11/11 (109 assertions, flags + the parser's upstream refusals + the builtin-only reach of `extra="forbid"`) + `test_chat_mm` 11/11 (126) + `test_openai_api_server` 56/56 (CASES; its assertion count is timing-dependent — 632/648/651 across three runs of one binary, so only the case count is quotable — the HTTP 400 arm proven against BOTH a 500 and a truncated 200); RED-first behavioural: `CHECK(500 == 400)`, `CHECK("InternalServerError" == BadRequestError)` and `CHECK(200 == 400)`; mutations proven RED: flag→config plumbing dropped, the call-site wiring dropped, and the refusal re-typed off `InputValidationError` (which lands as the 500 L1's design avoided). L4: `tests/scripts/test_check_oracle_denominator_flags.py` 11/11, RED-first behavioural (the checker on the pre-L4 tree exits 1 naming `dgx-online-serving.sh:487` and `:498` of 3 discovered launches) and mutation-proven (removing the flag from the canonical driver in a scratch copy returns exit 1 with exactly one violation, the exempt q3mxfp4 arm staying exempt) | [multimodal-track.md](specs/multimodal-track.md) §3 (M0/M1) | `READY` | - | | `ENG-MM-VISION-TOWER` | **Qwen3-VL vision tower `Qwen3_VisionTransformer` (M2a), proven faithful vs vLLM 0.25.0 in isolation.** The reusable vision half of the whole Qwen3-VL family + Qwen3.6 (27B/35B share this exact tower). Pure-additive C++ forward composed from public vt:: ops: patch-embed (Conv3d-as-matmul + bias), host pos-embed bilinear-interp+spatial-merge-reorder, 24 ViT blocks (LayerNorm + vision attention with partial-rotary NeoX vision RoPE via `vt::RopeFromCache` + non-causal `vt::Attention(causal=false)` + tanh-GELU MLP), patch merger (LayerNorm + exact-erf-GELU + 2 FCs), DeepStack 3 post-shuffle-norm mergers at layers 5/11/17 → `[196,10240]`. Adds 2 additive elementwise vt ops (`GeluTanh`/`GeluErf`). NO runner/model/registry edit → text engines byte-identical by construction. Proven faithful in ISOLATION; the merge into `input_embeds` + the MRoPE/DeepStack text backbone + the e2e image gate are M2b/M2c. | T1 | `vllm/model_executor/models/qwen3_vl.py` `Qwen3_VisionPatchEmbed:347`, `Qwen3_VisionBlock:413`, `Qwen3_VisionPatchMerger:467`, `Qwen3_VisionTransformer:519`, `forward:800`, `pos_embed_interpolate_native:277`, `rot_pos_emb:667`; `qwen2_5_vl.py::Qwen2_5_VisionAttention.forward:397`; `rotary_embedding/common.py::ApplyRotaryEmb.forward_static:151` @ `e24d1b24` | `src/vllm/model_executor/models/qwen3_vl_vision.{h,cpp}`; 2 vt ops `include/vt/ops.h` + `src/vt/ops.cpp` + `src/vt/cuda/cuda_layernorm.cu` + `src/vt/cpu/cpu_layernorm.cpp`; dumps `scripts/mm/m2a_tower_{ref,weight}_dump.py`; fixtures `tests/vllm/multimodal/fixtures/qwen3vl_tower/` | `tests/vllm/multimodal/test_qwen3vl_tower.cpp` — 4 RED-first tower gates vs the dumped vLLM-0.25.0 reference 348/348 (patch-embed 2.1e-3, block0 6.8e-3, merger 6.5e-2, DeepStack 1.2e-2/3.3e-2/4.4e-2, full tower 5.1e-2; pos-embed 2.5e-3 + rope 1.9e-3 TIGHT); bf16-depth envelope RCA'd; RED = rope disabled → block0 0.149/tower 0.75/6 fails; cutlass-ON+FA2 banner; clean `-Werror`; compute-sanitizer 0 — anchor `tests/vllm/multimodal/test_qwen3vl_tower.cpp:96` | [multimodal-track.md](specs/multimodal-track.md) §3 (M2a) | `ACTIVE` | `CLAIM-MULTIMODAL-M2A` | | `ENG-MM-TEXT-BACKBONE` | **Qwen3-VL text-backbone numeric contracts `Qwen3VLGetRopeIndex`/`Qwen3VLMergeMultimodal`/`Qwen3VLComputeDeepstack` (M2b/M2c), unit-green vs vLLM 0.25.0.** The deterministic pieces that fork the plain Qwen3-dense text path for a vision-conditioned decode: (1) MRoPE 3-D `get_rope_index` positions [3,T] (image tokens get (t,h,w) grid positions, text sequential); (2) the 3-section MRoPE APPLICATION — proven to be the EXISTING `vt::RopeFromCache` mrope path (positions [3,T] + `mrope_section=[24,20,20]` interleaved), faithful to `MRotaryEmbedding.forward_native` for Qwen3-VL's exact config; (3) `_compute_deepstack_embeds` scatter → [L,T,H] decoder-injection tensor; (4) `_merge_multimodal_embeddings` masked scatter of the tower's `[:,:2560]` into `input_embeds`. Pure-additive TU — NO shared dense forward / runner / registry edit → text engines byte-identical by construction. The e2e image forward (VL weight loader + forked MRoPE/DeepStack decode loop) is the remaining M2c wire-up. | T1 | `vllm/model_executor/models/qwen3_vl.py` `_get_mrope_input_positions:2567`, `_iter_mm_grid_hw:2482`, `_compute_deepstack_embeds:2761`, `Qwen3LLMModel.forward` deepstack `:1589`; `vllm/model_executor/models/utils.py::_merge_multimodal_embeddings:524`; `vllm/model_executor/layers/rotary_embedding/mrope.py` MRotaryEmbedding @ `e24d1b24` | `src/vllm/model_executor/models/qwen3_vl_text.{h,cpp}`; existing `vt::RopeFromCache` mrope path (`src/vt/{cpu,cuda}/*`); dump `scripts/mm/m2b_text_ref_dump.py`; fixtures `tests/vllm/multimodal/fixtures/qwen3vl_text/` — anchor `src/vllm/model_executor/models/qwen3_vl_text.cpp:9` | `tests/vllm/multimodal/test_qwen3vl_text.cpp` — 4 RED-first gates vs the dumped vLLM-0.25.0 reference 85/85 (get_rope_index BIT-exact [3,204], delta −182; MRoPE q rel-L2 1.5e-3 / k 1.5e-3, RED interleaved-off >5e-2; DeepStack + merge BIT-exact); CPU-only, no weights; clean CPU `-Werror` — anchor `tests/vllm/multimodal/test_qwen3vl_text.cpp:99` | [multimodal-track.md](specs/multimodal-track.md) §3 (M2b/M2c) | `ACTIVE` | `CLAIM-MULTIMODAL-M2BC` | | `ENG-MM-QWEN36-VL-FORWARD` | **Qwen3.6-27B (`Qwen3_5ForConditionalGeneration`) GDN-hybrid VL forward — IMAGE (M3-b) + VIDEO (M3d) BOTH e2e, STRICT gates PASS 32/32. Our own gate model's image+video paths now work end-to-end (speed pending).** The genuinely-new integration completing our own gate model's mm paths: fork the landed bf16 `Qwen3_5DenseModel` GDN-hybrid forward (48 GDN + 16 full-attn) on gated, default-off points so a text-only 27B request stays byte-identical — (a) `inputs_embeds` entry (embed ids + `Qwen3VLMergeMultimodal` scatter of the 27B tower merger `[N,5120]` into the visual-token rows; 27B has EMPTY `deepstack_visual_indexes` ⇒ NO DeepStack); (b) 3-section MRoPE (`mrope_section=[11,11,10]` interleaved, rotary_dim 64, theta 1e7) in the 16 full-attn layers only via the proven `vt::RopeFromCache` mrope path (GDN layers carry no rope); (c) mixed load = the M2a `Qwen3_VisionTransformer` (27B vision config, empty deepstack) bf16 tower + the bf16 GDN-hybrid LLM via the EXISTING `LoadQwen3_5Dense`. **M3d (2026-07-25) added VIDEO by REUSE:** the M3-b image driver refactored into a shared `VLGenerateCoreGdn`, image+video wrappers differ ONLY in the merge mask (`image_token` vs `video_token` across frames) + the get_rope_index (`Qwen3VLGetRopeIndex` vs `Qwen3VLGetRopeIndexVideo`); the M3c processor/windowed-tower/video-MRoPE are reused verbatim. | T1 | `vllm/model_executor/models/qwen3_5.py:389` (`Qwen3_5ForConditionalGeneration` subclasses `Qwen3VLForConditionalGeneration`; `visual = Qwen3_VisionTransformer`, modalities {"image","video"}); `qwen3_vl.py` `_process_video_input:2165`, `_get_mrope_input_positions:2567` video branch, `get_video_repl:1479`; the 27B `config.json` (`mrope_section=[11,11,10]`, empty `deepstack_visual_indexes`) @ `e24d1b24` / vLLM 0.25.0 | **M3-b + M3d BUILT + GATED 2026-07-25:** vision-only loader `LoadQwen3VLVisionWeights` (`src/vllm/model_executor/models/qwen3_vl.cpp`, 27B config) + shared `VLGenerateCoreGdn` + image driver `Qwen3_5VLGenerateGreedy` + **video driver `Qwen3_5VLGenerateGreedyVideo`** + `BuildMropeCosSinHost` + the `mrope_cos_sin` param on `DenseForwardLayers` (`src/vllm/model_executor/models/qwen3_5.cpp`, nullptr on every text caller ⇒ byte-identical; the video driver is purely additive, the shared text forward UNTOUCHED per `git diff --stat`) reusing M2a tower + `LoadQwen3_5Dense` bf16 LLM | **IMAGE:** golden `tests/vllm/multimodal/fixtures/qwen3_5_27b/` (STRICT sha256 `ead4b484…`); STRICT image gate PASS **32/32** (`test_qwen3_5_vl_e2e.cpp`, 54/54, re-run post-refactor). **VIDEO (M3d):** oracle `scripts/mm/m3d_video_oracle_capture.py` on the M3c synthetic clip (raw sha `8a111599…`, grid `[4,8,8]`, 64 video tokens) K=5 DETERMINISTIC ⇒ STRICT golden; **STRICT video gate PASS 32/32** (`test_qwen3_5_vl_video_e2e.cpp`, 27/27; near-tie gaps 0.0000 nats everywhere), fixtures `tests/vllm/multimodal/fixtures/qwen3_5_27b_video/`. Text-inertness 27B 235/235, 35B 315/315, Coder 138/138 (by construction); clean `-Werror` 0 warn; compute-sanitizer 0 on the 27B video forward. **SPEED MEASURED (2026-07-26, `CLAIM-MULTIMODAL-SPEED`): image c1 vs vLLM 0.25.0 GRAPHED — decode TPOT 225.0 ms/tok vs 226.9 = AT PARITY (0.99×), LLM prefill 326 ms vs vLLM TTFT 321 ms = at parity; vision tower WAS 2114 ms vs vLLM encode ≤~250 ms = ~10× (THE gap). TOWER LEVER EXECUTED (2026-07-26, `CLAIM-MULTIMODAL-SPEED-TOWER`, [multimodal-speed.md](specs/multimodal-speed.md) §7): nsys `cuda_gpu_kern_sum` attributed 98.9 % of the tower forward to the naive `vt::cuda::AttentionKernel` (56 ms/block; NOT QKV/FA2-routing); fixed by a warp-scoped online-softmax op `AttentionDenseFast` (separate op ⇒ `kAttention`/text byte-identical) + one-time resident-weight load ⇒ per-image tower 2114 → 148 ms (14.3×), **0.59× vs vLLM eager encode = FASTER**. STRICT image/video e2e HELD 32/32 (+4B DeepStack 32/32), `test_ops_attention` 37239/37239, 27B text SACRED 235/235, compute-sanitizer memcheck 0, clean `-Werror`. `benchmark_binding=false`, single-seq driver (no c2+/server). Remaining: batched/graphed mm serving (c2+) + audio our-side — DONE bar not yet met.** | [multimodal-track.md](specs/multimodal-track.md) §M3 + [multimodal-speed.md](specs/multimodal-speed.md) §7 + §8 (decode lever #2 CLOSED 2026-07-27: on-GPU greedy argmax + decode embed round-trip removed on `VLGenerateCoreGdn`; bit-exact — image/video STRICT 32/32 held; 27B decode NEUTRAL at the ~222 ms bandwidth floor) + §9 (lever #3 FIRST BRICK 2026-07-27, `CLAIM-MULTIMODAL-SPEED-GRAPH`: the shared `VLGenerateCoreGdn` decode step now routes through the production `Qwen3_5DenseDecodeGraph` cold→warm→replay captured decode — the mm decode is now GRAPH-CAPTURABLE, closing the un-graphed-eager-loop structural gap; S==B==1 bit-identical rebuild; token-exact HELD image/video STRICT 32/32 with 30 graph replays confirmed; A/B graphed 232.5 vs eager 233.4 ms/tok = NEUTRAL at the 27B bandwidth floor; the launch-overhead win + batched c2+ + serving ingestion are the recorded W-plan W1-W3) + §16 (vision-forward flash kernel 2026-07-28, `CLAIM-MM-SPEED-QWEN-IMAGE`: ATTRIBUTION-FIRST nsys attributed ~85% of the 148 ms tower forward to the dense attention `AttentionWarpKernel` [4.66 ms/block×27]; routed it to the §14 flash-tiled `vt::AttentionDenseFlash` [head_dim 72, byte-identical — per-warp math verbatim, only K/V from shared-mem tiles]. STRICT image/video e2e HELD 32/32 [27B+4B], `test_ops_attention` 37239/37239, goldens md5 UNCHANGED, nsys proof `AttentionDenseFlashKernel` 24 inst/zero warp, RED 30/46→46/46, sanitizer 0. A/B warp 148.3→flash 142.3 ms = 1.04× — the profile REFUTED a big lever: at t=784 the vision attention is serial-latency-bound not bandwidth-bound [audio §14 was 1.82× at t=1500], flash recovers only ~6 ms. **HONEST: the tower ALREADY BEATS vLLM — 142 ms vs ~250 ms eager encode = 0.57×**; image/video mm-forward is correctness-DONE + speed-BEATS-vLLM; residual = tensor-core MMA hd-72 attention [not needed for parity] + batched c2+/serving) | `ACTIVE` | `CLAIM-MULTIMODAL-SPEED-TOWER` + `CLAIM-MULTIMODAL-SPEED-DECODE` + `CLAIM-MULTIMODAL-SPEED-GRAPH` + `CLAIM-MM-SPEED-QWEN-IMAGE` | diff --git a/.agents/issue-index.md b/.agents/issue-index.md index e8017c3f0..3dd10cef2 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -433,6 +433,8 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#1325](https://github.com/mudler/vllm.cpp/issues/1325) | `ENG-RECORD-ANCHOR-RATCHET` | `scripts/record-anchor-baseline.json` stores a `"total"` that no code reads. `load_record_anchor_baseline` in `scripts/check-agent-record.py` returns `{bucket: int(data["buckets"][bucket]) for bucket in RECORD_ANCHOR_BUCKETS}` with `RECORD_ANCHOR_BUCKETS = ("stale", "broken")`, and it is the file's only reader; `check_record_anchors` iterates those two buckets, and `write_record_anchor_baseline`'s refusal compares `result.total > sum(previous.values())` — the buckets, not the stored `total`. Measured at `af87251c5`: mutating `"total": 38` to `39` (a file whose total disagrees with `32 + 6`) leaves `check-agent-record.py` exit 0, tree restored byte-for-byte by sha256. So this row's own budget file carries the exact shape the row exists to name: a recorded figure no gate reads, sitting beside the figures that are read and presenting as if it were checked. `--write-baseline` compounds it by printing `-> 38`, which reads as the value it stored and is the one no later run consults. TWO candidate resolutions, deliberately not chosen here because choosing belongs to the fixing row: read it and assert `total == stale + broken` on load, or drop the field and derive it at read time, which is the shape AGENTS.md §Records prefers. Either is a semantic change to `check-agent-record.py` owing a spec and a red-before case in `tests/scripts/test_agent_record.py` `RecordAnchorRatchet`. Distinct from [#1287](https://github.com/mudler/vllm.cpp/issues/1287) and [#1270](https://github.com/mudler/vllm.cpp/issues/1270), neither of which reaches the unread field. Also under `## Owed` in [`record-anchor-ratchet.md`](specs/record-anchor-ratchet.md) | bug | | [#1316](https://github.com/mudler/vllm.cpp/issues/1316) | — | `scripts/main-baseline.py` renders a scheduled run that executed ZERO jobs as `RED` with all 11 covered jobs `missing`, so `NEWEST BASELINE: RED at ` names a tree the run never checked out. Measured at `origin/main` `250db75a2`: runs `32206456661` and `32140419182` both return `startedAt: null` for every job, because GitHub cancelled them while they were pending in the single `ci-schedule-refs/heads/main-mudler/vllm.cpp` group, whose queue holds one run ([#274](https://github.com/mudler/vllm.cpp/issues/274)). Fail-closed, and the `missing (expected, never ran)` line is accurate about the jobs; the defect is the verdict word, because a run that executed nothing is NOT RUN rather than RED, and the newest verdict should fall through to the newest run that actually ran. NOT fixed in flow: `test_an_expected_job_the_payload_never_mentions_is_red` and `test_a_narrowed_run_reports_red_and_names_what_never_ran` deliberately assert missing-is-red so a narrowed run cannot pass, and separating "narrowed" from "never started" changes what the verdict means, which owes its own spec, red-before evidence and a fresh reviewer. Owed under `## Owed` of [baseline-lane-eviction.md](specs/baseline-lane-eviction.md), which removes the only observed producer of a zero-job run | bug | | [#1314](https://github.com/mudler/vllm.cpp/issues/1314) | `SPEC-DFLASH2` | **DFlash2 (`DFlash2DraftModel`) is unported**, and one config rule would run the published checkpoint wrong in silence. Upstream carries DFlash2 as a SECOND architecture beside DFlash rather than as a change to it ([vllm#52816](https://github.com/vllm-project/vllm/pull/52816), OPEN at head `19c9351904df4c63042671bc67a866ca48dc7d6f`, base `9842d701`, 755+/5-, 11 files, plus the stacked guard fix [vllm#52883](https://github.com/vllm-project/vllm/pull/52883)): DFlash1 gains two subclass seams and keeps every behaviour, and the new architecture adds a GROUPED DYNAMIC DEPTHWISE CONVOLUTION around each attention and each MLP sublayer plus a CANDIDATE SELECTOR that replaces the independent per-slot argmax with a scored path walk over the target head's top-K. Shapes taken from the published checkpoint rather than from the diff: `z-lab/Qwen3.8-27B-DFlash2`, safetensors header range-read 2026-08-19, 81 tensors -- DFlash1's set plus `layers.N.{attention,mlp}_conv.{base_kernel (2,2,5120), kernel_projection.weight (1280,5120)}` x5 and `candidate_selector.{hidden_projection.weight (256,5120), predecessor_codebook, successor_codebook}` at `(248320,256)` bf16 each, ~254 MB resident the DFlash1 lane never allocates; `conv_kernel_size 2`, `conv_group_size 16`, `selector_rank 256`, `selector_top_k 16`, `block_size 8`. **The silent one:** that config declares all five layers `sliding_attention` AND `is_causal false`, while our resolution mirrors the OLD upstream rule (causal iff SWA, unless `dflash_config.causal`, `include/vllm/model_executor/models/qwen3_dflash.h:22-24`), so every layer would run CAUSAL -- plausible tokens, a token gate against our own output sees nothing, and only ACCEPTANCE moves, which the lossless verify hides. Upstream changes `_dflash_layer_causal` to read `is_causal` first, in the same commit. Three further things are owed and none of them is silent: no route for the `DFlash2DraftModel` architecture string (the same classification code as the open `DSparkDraftModel` gap, [#1193](https://github.com/mudler/vllm.cpp/issues/1193)); no top-k that EMITS the surviving (id, value) pairs, where the decision is to extend the sort-free pivot-bracket threshold search already ported from the same FlashInfer approach at `src/vt/cuda/cuda_sample.cu:297-506` rather than port FlashInfer's 3380-line general radix kernel; and the path walk must run ON DEVICE from the first landing, because the identical sequential shape in DSpark shipped host-side and measured 28% of the 27B draft step ([#436](https://github.com/mudler/vllm.cpp/issues/436)) before `SampleSequentialDevice` moved it. Already reusable unchanged: `vt::DFlashBlockAttention`, the DFlash runner/rejection/GDN-rollback lane, and the loader's target-shared `embed_tokens`+`lm_head`, which is already what a DFlash2 checkpoint needs. BEYOND-PIN by developer decision 2026-08-19 (mirror the open PR now, reconcile if review moves it), in the same posture `SPEC-DSPARK-QWEN3-ROUTING` takes toward vllm#52197; the parity pin `555967922` does not carry the architecture and is NOT advanced. Gate arm is bf16 27B plus the GGUF drafter arm in the same wave, oracle = vLLM built at the PR head, acceptance measured SAME-TRAJECTORY because `SPEC-DFLASH` D8 spent a campaign on a divergent-trajectory confound that D9 refuted. Spec [dflash2-spec-decode.md](specs/dflash2-spec-decode.md) | feature | +| [#1340](https://github.com/mudler/vllm.cpp/issues/1340) | `ENG-MM-INPUT-PIPELINE` | `VT_FUSE_ATTN_PREAMBLE=0` on the Qwen3.6 MRoPE path silently applies 1-D RoPE instead of refusing. Our fused preamble is the ONLY MRoPE arm: `vt::AttnQkNormRopeGate` takes no positions at all, only a precomputed per-token `cos_sin` cache `[T, rotary_dim]`, which `src/vllm/model_executor/models/qwen3_5.cpp::BuildMropeCosSinHost` fills with the interleaved 3-section MRoPE axis selection for the M3-b image and M3d video paths. The eager arm in `::FullAttnBlockPaged` calls `vt::RopeNeox` on the 1-D positions vector, which that same file comments as "unused for rope under the fused MRoPE path", so with the toggle off the model computes plain NeoX RoPE where MRoPE is required: every shape agrees, nothing throws, the tokens are wrong. AGENTS.md requires an unimplemented arm to be REFUSED naming the missing part, and this one is substituted instead. Found while landing [#607](https://github.com/mudler/vllm.cpp/issues/607) wave L4, which is the same asymmetry read the other way: mirroring upstream's `text_only` conjunct (`vllm/model_executor/models/qwen3_next.py:324-331` at the pin) would route the multimodal configuration into this arm deliberately. NOT fixed in flow because its gate is a VL token-exactness run through `ModelRegistry::Forward` on a real checkpoint and the L4 wave had no GPU: both fleet devices were held and it carried no lease authority. Also under `## Owed` in [`multimodal-track.md`](specs/multimodal-track.md) §1.6 | bug | +| [#1345](https://github.com/mudler/vllm.cpp/issues/1345) | `ENG-MM-INPUT-PIPELINE` | [#414](https://github.com/mudler/vllm.cpp/issues/414) on a second surface: three in-process bench harnesses construct the vLLM oracle with `language_model_only` left at its `False` default and expose no way to set it -- `tools/bench/profile_vllm_online_gate.py:209` (the online gate's torch-profiler arm), `tools/bench/vllm_closed_loop_metrics.py:160` (the reference companion to `examples/bench/bench_core.h`) and `tools/bench/dump_vllm_tokens.py:34` (greedy oracle token capture). `EngineArgs.language_model_only` is an ordinary `LLM(...)` keyword (`vllm/engine/arg_utils.py:555,1691` at the pin), so pointed at a Qwen3.6 checkpoint each gets `text_only == False` and the UNFUSED QK-norm+RoPE+gate path against our fused one. For the two measurement harnesses that is the flattered denominator; for the token dump the direction is not automatic, because the two arms are not bit-identical and a golden captured on one can sit on the far side of a near-tie from the arm it gates. Deliberately OUTSIDE `scripts/check-oracle-denominator-flags.py`, which [#607](https://github.com/mudler/vllm.cpp/issues/607) wave L4 landed: all three take `--model` as a path, so no static rule can know whether a run points at a multimodal checkpoint, and a checker that cannot decide its own question is worse than none. Fix is to thread the knob through and record the RESOLVED value beside the measurement; the default is a denominator decision for the operator. NOT fixed in flow because it changes what three harnesses hand the oracle and the L4 wave could take no measurement to exercise it once. Also under `## Owed` in [`multimodal-track.md`](specs/multimodal-track.md) §1.6 | bug | | [#1342](https://github.com/mudler/vllm.cpp/issues/1342) | `VT-ACT-ROUND-POLARITY` | `src/vt/vulkan/shaders/vt_silu_and_mul.comp` and `src/vt/metal/metal_msl.h` both carry a comment claiming the silu math is "ported 1:1 from `src/vt/cpu/cpu_ops.cpp` SiluAndMulKernel, including its `gate / (1 + exp(-gate))` spelling", and both emit `gate * vt_sigmoid(gate) * up` instead, where `vt_sigmoid(x) = 1.0/(1.0+exp(-x))` (`vt_common.glsl`). The reciprocal is rounded before the multiply, so it is one extra f32 rounding step that the CPU and CUDA kernels do not take; upstream `activation_kernels.cu::silu_kernel` spells it as the division too. `vt_rms_norm_gated.comp` warns against precisely this form in its own header, so the convention is written down and these two files are on the wrong side of it. Invisible to every gate we own, because both forms are single-rounding-on-store and a bf16-eps parity test passes either way — the false part is the citation. Found during the #1322 inventory. NOT fixed in flow: it is a separate defect from the polarity gap, survives that fix untouched, and belongs to whichever row repairs the two shaders. Owed under `## Owed` of [vt-act-round-polarity.md](specs/vt-act-round-polarity.md) | bug | | [#1343](https://github.com/mudler/vllm.cpp/issues/1343) | `VT-ACT-ROUND-POLARITY` | `vt::RmsNormPlusAdd` computes different numbers on different devices for a bf16 output. The ROCm arm (`src/vt/rocm/rocm_rmsnorm.hip`) keeps the whole chain in one f32 expression — `Store(orow, j, Load(xrow, j) * inv * wj + Load(arow, j))` — while the composed arm in `src/vt/fused_ops.cpp` calls `RmsNorm(q, out, x, w, args)` then `Add(q, out, out, addend)`, materializing `out` between them so the normalized value is rounded to bf16 BEFORE the addend is added. One `vt::` entry point, two numeric behaviours, selected by device. That is the shape AGENTS.md §Shared seams exists to prevent, and the project already holds `vt::Conv1d`/`ConvTranspose1d` to byte-identical host/CUDA providers as the standard. Invisible to a cross-device parity test with a bf16-eps tolerance, because the difference is at most 1 bf16 ULP per element. Found during the #1322 inventory. NOT fixed in flow: choosing the direction needs its own reading of the upstream fusion — either the composed arm grows a fused kernel that keeps f32 across the add, or the ROCm arm rounds — and only one matches upstream, so it owes a spec, a red-before test a tolerance cannot satisfy, and a fresh review. Owed under `## Owed` of [vt-act-round-polarity.md](specs/vt-act-round-polarity.md) | bug | | [#1334](https://github.com/mudler/vllm.cpp/issues/1334) | `MODEL-MUSIC-minimax-music3-mini-max-music3-for-conditional-generation` | MiniMax-Music3's vocoder is **53.6 s of a 161.6 s run (33.2 %)** after [#1238](https://github.com/mudler/vllm.cpp/issues/1238), against 12.0 % before it, and #1238's depth A/B measured it at 53.6 s on BOTH legs — it is a FIXED term that grows as a share of every other improvement. **Both arms are slow for one cause.** `vt::cpu::Conv1dKernel` (`src/vt/cpu/cpu_conv1d_general.cpp::Conv1dKernel`) computes each output cell with ONE f64 accumulator swept over `(ic ascending, k ascending)`, which for a Music3 residual unit is `384 * 7 = 2688` STRICTLY DEPENDENT f64 additions per output element: no instruction-level parallelism, no vectorisation at any width, and a measured 1.7-2.0 GMAC/s per core = ~2.8-3.0 cycles per multiply-accumulate on a 5.0 GHz Zen 5 whose `fadd` latency is 3. The CUDA provider loses by a DIFFERENT mechanism with the same cause (§15.9: 3.552 s device vs 2.983 s host at latent length 20; §13.10's per-stage ratios flat at 0.37-0.40x across a 150x span of work): one accumulator per cell means it is not latency-bound but f64-RATE bound, and Thor's consumer Blackwell runs fp64 at a fraction of its fp32 rate. **The f64 stays.** §13.2 records it as what all four consumers' goldens were taken with and what makes the CUDA provider `memcmp`-identical; narrowing to torch's f32 re-gates four shipped models and is left OWED. **The chain breaks bit-identically instead**: one f64 accumulator per cell over a TILE of output positions with the `(ic, k)` sweep hoisted outside, so every cell receives the identical sequence of IEEE-754 double additions of the identical products in the identical order, interleaved across independent cells rather than serialised — a scheduling change, not an arithmetic one, so `memcmp` survives by construction. Measured single-threaded at the real vocoder shapes, `-ffp-contract=off`, `memcmp`-identical on every case: **3.4x-6.5x** per stage. Spec [`minimax-music3.md`](specs/minimax-music3.md) §18 | perf | @@ -456,6 +458,7 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#1366](https://github.com/mudler/vllm.cpp/issues/1366) | `SPEC-DFLASH2` | DFlash causality diverged from upstream on the `use_swa` arm: the legacy fallback read the RESOLVED `is_sliding`, which `dflash_config.use_swa` forces true, while upstream reads the RAW `layer_types` (`bool(layer_types) and layer_types[i] == "sliding_attention"`, `qwen3_dflash.py:58-67` @ vllm#52816 head `19c93519`). Upstream's own `_resolve_layer_attention` docstring table states `layer_types=None` + `use_swa=True` -> causal False and names `XiaomiMiMo/MiMo-V2.5-Pro-FP4-DFlash` as that shape, so such a DFlash1 draft ran EVERY layer causal here and non-causal upstream. Acceptance-only and token-gate-invisible, because the verify is lossless. Uncaught because upstream's parametrize table has no `use_swa` row either, so the ported cases were faithful to the ported table and silent about the arm. Second divergence in the same resolution: `is_causal` was honoured only when `.is_boolean()`, while upstream tests presence and coerces (`if is_causal is not None: return bool(is_causal)`), so `"is_causal": 0` fell through in silence -- and the GGUF arm was already more permissive through `KvI64`, so the two containers disagreed with each other as well as with upstream. Found by the fresh reviewer of `SPEC-DFLASH2` W1 and fixed IN FLOW on `row/SPEC-DFLASH2-W1`, red-first: the two docstring rows redden 4 assertions before the repair | bug | | [#1370](https://github.com/mudler/vllm.cpp/issues/1370) | `ENG-HF-MODEL-DOWNLOAD` | `HubListRepoFiles`'s tree-listing size rule was evadable by LETTER CASE. `IsHexString` (`src/vllm/transformers_utils/hf_hub.cpp:30`) accepts `A`-`F` as well as `a`-`f`, mirroring llama.cpp's `is_valid_oid` (`common/hf-cache.cpp:161 @ b10451`), and `oid_owner` is a `std::map` keyed on the RAW identifier with nothing folding case anywhere in the file. Two spellings of one identifier landed TWO keys and the rule compared nothing. MEASURED at `db0af37b1`: `a.bin` at 4096 bytes under `ab234567...` beside `b.bin` at 2048 bytes under `AB234567...` is **ACCEPTED**, where the same pair in one case is refused. This is [#1339](https://github.com/mudler/vllm.cpp/issues/1339) through a different door: `HF_ENDPOINT` is user configurable, so the listing is input the hub does not have to answer truthfully, and where one omitted `size` field bought an unconditional pass there, one changed letter case bought the same pass here. The raw-identifier assumption reaches PAST the owner map: `HfFile::oid` left the function in the listing's case and `HfBlobPath` (`src/vllm/transformers_utils/hf_cache.cpp:145`) makes that value a cache FILE NAME, so one listing would populate a different cache on a case-sensitive file system than on a case-insensitive one such as this project's CIFS checkpoint mounts. That half is latent, because `HfBlobPath` has no production caller until W3. `IsDegenerateOid` is unaffected, since folding a repeated character leaves it repeated. FIXED IN FLOW during the fourth fresh review of [#1282](https://github.com/mudler/vllm.cpp/pull/1282): the identifier is folded once, on the identifier ITSELF rather than only on the map key, after the hexadecimal form is validated and before either rule or `HfFile::oid` reads it. FOLDED rather than REFUSED, because hexadecimal is case-insensitive by definition and both git and the hub emit lower case, so an upper-case spelling is another spelling of one value and refusing it would reject a mirror over a difference that means nothing. Four cases RED first: the mixed-case pair at disagreeing sizes, the fold reaching `HfFile::oid`, the mixed-case pair at an agreed size still accepted, and an upper-case degenerate identifier still refused. Spec [`hf-model-download.md`](specs/hf-model-download.md) | bug | | [#1353](https://github.com/mudler/vllm.cpp/issues/1353) | `ENG-RECORD-ANCHOR-RATCHET` | A full disk makes `scripts/agent-preflight.sh` report record and policy defects that do not exist. Measured at `63ff58272` with 896M free of 447G: ten suites go red together — `test_check_release_binary_contract`, `test_release_postpublish_audit`, `test_check_container_matrix`, `test_release_index`, `test_release_metadata`, `test_release_accelerator_metadata`, `test_release_macos_metadata`, `test_release_windows_metadata`, `test_agent_role`, `test_agent_onboard` — and the trailing summary names all ten without mentioning the disk. The cause appears only inside one suite's captured output several screens up (`error: copy-fd: write returned: No space left on device`), interleaved with fixture failures that read exactly like findings (`ERROR: x: see (#157) and #174`, `ERROR: x: landed without a row PR`). All ten build a scratch git repository in a temporary directory, so all ten fail together and all ten fail toward a verdict about RECORDS rather than about the environment — the `.agents/verification.md` broken-instrument shape, and an agent reading it has every reason to start repairing records that were never broken. Reclaiming 2.7G of one row's own build tree and re-running the identical command turned all ten green with no tree change (`86 gates, 86 ok, 0 FAIL, 0 SKIP`, exit 0). The fix is a precondition, not a suppression: check free space before the suites that need a scratch repository and refuse naming the disk, exactly as `test_cpu_x86_llamacpp_floor` refuses to measure under contention ([#618](https://github.com/mudler/vllm.cpp/issues/618)). A gate that cannot run must say so rather than return a verdict. Found while landing [#1332](https://github.com/mudler/vllm.cpp/issues/1332) M0+M1 and NOT fixed in that flow, because it changes preflight semantics and adds a refusal path, which `AGENTS.md` routes to the normal row, spec and fresh-review path | bug | +| [#1373](https://github.com/mudler/vllm.cpp/issues/1373) | `ENG-RECORD-CONFLICT-SURFACES` | `.agents/benchmark-record.md:3` self-declares "Append-only forensic record" but `.gitattributes` gives `merge=union` only to `.agents/issue-index.md` (`.gitattributes:7`), and the file is 7 lines long so nothing else supplies one. Two append-only record surfaces, both declared append-only in prose, and the union driver on one of them. AGENTS.md §Records admits "a genuinely append-only file that can union-merge" as a record shape and warns that "if N concurrent pull requests edit file F, that file is a lock"; without the attribute this file IS that lock, because two branches that each append a section collide on the same tail with zero overlapping content. MEASURED on [#1349](https://github.com/mudler/vllm.cpp/pull/1349) in one afternoon: merging `9e1a5e573` (#1369) gave `CONFLICT (content): Merge conflict in .agents/benchmark-record.md`, and merging `2f67c9358` gave the same conflict again as the ONLY conflict once `docs/BENCHMARKS.md` had settled — both pure tail appends on both sides (49 lines ours, 393 and 234 theirs), each costing a full resolve/re-verify/re-gate cycle, the second losing a merge race. SCOPE for the fixing row, so it does not over-promise: per [#883](https://github.com/mudler/vllm.cpp/issues/883) the union driver is LOCAL, so GitHub will still report these appends as conflicting in its own UI; what it buys is that `git merge` and `git merge-tree` stop conflicting locally, which is where the cycles are spent. Also owed there, not here: whether `scripts/roll-benchmark-record.py` can produce a NON-tail edit, since union-merge is only sound while every write is genuinely an append. NOT fixed in flow — changing a merge driver changes how every future concurrent append resolves, which is a semantic change to the repository's conflict behaviour and wants its own spec and fresh review rather than a one-line ride-along in a multimodal PR | bug | | [#1374](https://github.com/mudler/vllm.cpp/issues/1374) | `ENG-CUDAGRAPH-BREAK` | W6, the last stage: the graph-eligibility predicate moves out of the two model files that each re-derived it and into `GPUModelRunner::execute_model`, which names the step's ACTUAL uniform query length once through `v1::GraphEligibleQueryLen` and ships it on `ModelForwardInput::uniform_query_len`. That closes [#1020](https://github.com/mudler/vllm.cpp/issues/1020) together with a `(S, q, spec)` ring key, which was a LIVE collision rather than the enabler #1020 called it. **What did NOT move is 'except at the break points'**: no driver in this tree serves a prefill or a mixed batch under any predicate, so that needs a prefill capture driver nobody has written and whose benefit the spec's D5 already refutes on this hardware — recorded as a publishable negative in `## Owed` rather than as a stage that ran out of window | feature | | [#1380](https://github.com/mudler/vllm.cpp/issues/1380) | `ENG-CUDAGRAPH-BREAK` | A speculative decode-graph capture does a `cudaMalloc` inside the captured region and throws on `thor:gpu0` (sm_110), and the queue is POISONED afterwards. Located to the SECOND parity-ring slot: slot 0 cold, slot 1 cold, slot 0 captures (`captured()` true, `replay_count()` 1), slot 1's capture throws `cudaMalloc: operation not permitted when stream is capturing`, and the next step fails with `embedding: operation failed due to a previous error during capture` without opening a scope. So a REPLAY is unreachable on a speculative shape there, on a default-ON path (`VT_SPEC_DECODE_GRAPH`). The driver's own pre-grow names this case and covers only the retained `[S, vocab]` logits. Found by [#1374](https://github.com/mudler/vllm.cpp/issues/1374) and PRE-EXISTING — the case drives the driver directly, bypassing the predicate, and the five migrated drivers read 2066 assertions / 0 differing on the same binary. NOT fixed in flow: it is a device-level allocation defect on a path W6 did not write, it needs `dgx`/sm_121a and a real checkpoint to scope, and `AGENTS.md` routes a surprising fix to the normal row, spec and fresh-review path. Owned by row `ENG-CUDAGRAPH-BREAK`, under `## Owed` in [`eng-cudagraph-break.md`](specs/eng-cudagraph-break.md) | bug | | [#1376](https://github.com/mudler/vllm.cpp/issues/1376) | `ENG-CUDAGRAPH-BREAK` | `main` was red on `tests/scripts/test_check_gate_commands.py`, measured at `601b576c6` in a detached worktree of `origin/main`: 8 failures of 44 tests, every one a comparison between the computed runnable population and `RUNNABLE_BASELINE`. `ENG-CUDAGRAPH-BREAK` was in the first and absent from the second. Cause: W5 of that row ([#1361](https://github.com/mudler/vllm.cpp/issues/1361)) filled its spec's `## Gates` section with runnable evidence, including a named test binary with its case and assertion counts and an exit status, which is exactly what moves a row into the runnable population. The ratchet's own error text instructs a re-pin in the SAME change, and the re-pin was not made. This is the growth case the ratchet exists to force a decision about, not a defect in that row's work. **It landed with no remote verdict**: the continuous integration lane that would have caught it independently has not executed for this repository since roughly 07:43Z on 19 August 2026, with runs queueing and none starting while GitHub reports Actions operational. FIXED IN FLOW while merging `origin/main` into `row/ENG-HF-MODEL-DOWNLOAD` for [#1280](https://github.com/mudler/vllm.cpp/issues/1280), because the fix is small and clear and a red `main` blocks every other row's gate. The entry is added with a justifying comment in the form the neighbouring entries use, no checker semantics change, and no test is weakened. After the re-pin the suite reports 45 tests OK and the audit reads 39 runnable of 119 gated rows | bug | diff --git a/.agents/specs/multimodal-track.md b/.agents/specs/multimodal-track.md index a677da43a..e2e4e2c12 100644 --- a/.agents/specs/multimodal-track.md +++ b/.agents/specs/multimodal-track.md @@ -458,8 +458,9 @@ comparing the two arms must set the flag on both sides or state that it did not. - **L3** — the tower skip: construct-without-initialising when every limit is 0, gated on **measured** RSS reduction against a multimodal checkpoint, plus token-exactness of the text path with and without the flag. -- **L4** — mirror the kernel gate, or record an explicit tracked exception with - the #414 cross-reference. +- **L4** — the kernel gate. **RESOLVED 2026-08-19 as a TRACKED EXCEPTION, not a + mirror (#607, cross-reference [#414](https://github.com/mudler/vllm.cpp/issues/414)).** + The full argument, anchors and gates are §1.6 below. **L2 is shippable without L3 only because L1 carries the refusal.** The first draft of this section said "L2 without L3 is honest and shippable: the flag @@ -477,6 +478,239 @@ all** — it is a flag that is accepted and inert, which is worse than the abort it replaces, because an abort is visible and a silently-served image request is not. +### 1.6 L4 — the kernel gate: why we do NOT mirror `text_only` (#607, #414) + +**Scope.** Decide whether our fused QK-norm + RoPE + gate preamble carries +upstream's `text_only` conjunct, record the decision, and make the benchmark +denominator that the divergence affects impossible to get wrong by accident. +In scope: the decision and its argument; the checker that pins the oracle +configuration every harness must launch; the harness reconciliation. OUT of +scope, each with a reason: the tower skip (that is L3, implemented separately); +any GPU measurement (both fleet devices were held, and no ratio below is +re-measured here); advancing the pin. + +**Upstream anchors, read at the pin `5559679229bc961848b121ccdeaa8fa5d79bec98`.** + +| upstream | file:line @ the pin | what it says | +|---|---|---| +| the flag | `vllm/engine/arg_utils.py:1275-1277` | `--language-model-only` over `MultiModalConfig.language_model_only` | +| the default | `vllm/config/multimodal.py:78` | `language_model_only: bool = False` | +| the predicate | `vllm/model_executor/models/qwen3_next.py:324-331` (`Qwen3NextAttention.__init__`) | `text_only = mm_config is None or mm_config.language_model_only`; `use_fused_qk_norm_rope_gate = attn_output_gate and is_neox_style and current_platform.is_cuda() and text_only` | +| the reason for the conjunct | `qwen3_next.py:323` | `# TODO: support MRoPE`, immediately above the predicate | +| the same reason, restated at the call | `qwen3_next.py:344-352` (`Qwen3NextAttention._project_qkv_gate`) | *"mRoPE passes positions as (3, n_tokens) for T/H/W. Fusion is only enabled text-only, where the three rows are identical, so taking the T row is exact."* — then `pos = positions[0] if positions.ndim == 2 else positions` | +| the kernel | `vllm/model_executor/layers/fused_qk_norm_rope.py:117-201` (`fused_qk_rmsnorm_rope_gate`) | takes `cos_sin_cache: (max_pos, rotary_dim)` **and** `positions: (n_tokens,)`, and indexes the cache by position inside the kernel | +| the eager arm | `qwen3_next.py:366-387` | split → `q_norm`/`k_norm` → `self.rotary_emb(positions, q, k)`, where `rotary_emb` **is** the MRoPE module for a multimodal Qwen3.5 | +| how Qwen3.5 reaches it | `vllm/model_executor/models/qwen3_5.py:146-153` (`Qwen3_5DecoderLayer.__init__`) | `full_attention` layers construct `Qwen3NextAttention(config, model_config=model_config, …)` | +| why our gate models land on `text_only == False` | `qwen3_5.py:399,614` (`Qwen3_5ForConditionalGeneration`, `Qwen3_5MoeForConditionalGeneration`) with `multimodal.py:78` | both read `vllm_config.model_config.multimodal_config`, which is non-`None`, so `text_only` is `False` unless the flag is passed | + +Every claim carried into this section from #414 reproduces at the pin. One +refinement to how it is usually restated: the platform conjunct is +`current_platform.is_cuda()` with **no** compute-capability test, so GB10 +`sm_121` is eligible — but `is_cuda()` is one of four conjuncts, not the whole +predicate, and the conjunct that decides our question is `text_only`. + +**Design — the decision, and why mirroring is not representable at our seams.** + +Upstream's predicate selects between two arms that BOTH compute correct RoPE. +Its fused Triton kernel derives RoPE by indexing `cos_sin_cache` with a 1-D +`positions` vector and therefore cannot express MRoPE; its eager arm calls +`self.rotary_emb(positions, q, k)`, which for a multimodal Qwen3.5 IS the MRoPE +module. `text_only` is upstream routing around its own acknowledged kernel gap — +the `# TODO: support MRoPE` sits directly above the predicate — and the gap is +invisible in behaviour because the arm it falls back to is complete. + +Our two arms are not those two arms. + +- `vt::AttnQkNormRopeGate` (`src/vt/ops.cpp::AttnQkNormRopeGate`, with + `src/vt/cuda/cuda_ops.cu::AttnQkNormRopeGateKernelCuda` and + `src/vt/vulkan/vulkan_ops.cpp::AttnQkNormRopeGateKernel`) takes **no positions + argument at all**. It consumes a precomputed per-token cache `cos_sin` + `[T, rotary_dim]` and reads row `t` for token `t`. Position semantics are + resolved by whoever fills the cache, before the launch. +- `qwen3_5.cpp::BuildMropeCosSinHost` fills that cache with the interleaved + 3-section MRoPE axis selection. Our M3-b image and M3d video paths inject it + verbatim through the `mrope_cos_sin` parameter of + `qwen3_5.cpp::VLForwardLayersFor` and its MoE twin. **Our fused arm IS the + MRoPE arm.** +- Our eager arm (`qwen3_5.cpp::FullAttnBlockPaged`, the `else` branch) calls + `vt::RopeNeox(…, sdi.positions, …)` on the 1-D positions vector. It is plain + NeoX RoPE. It is **not** MRoPE and has no MRoPE spelling. + +So the two predicates do not commute with the two dispatch tables. Conjoining +`text_only` onto `qwen3_5.cpp::FuseAttnPreambleOn` would, on exactly the +configuration the conjunct exists to protect, select an arm that computes 1-D +RoPE where MRoPE is required — silently, because every tensor shape still +agrees. It would break the landed image (M3-b) and video (M3d) STRICT 32/32 +gates, and it would do so in the name of mirroring a TODO. + +**Decision: TRACKED EXCEPTION.** We omit upstream's `text_only` conjunct from +`qwen3_5.cpp::FuseAttnPreambleOn`, deliberately and permanently, because our +fused kernel subsumes the capability whose absence motivates the conjunct +upstream. Owner: `ENG-MM-INPUT-PIPELINE` (this spec). Cross-reference: +[#414](https://github.com/mudler/vllm.cpp/issues/414). This is the exception arm +the L4 line offered, taken on the "cannot be represented at our seams" ground +AGENTS.md names rather than on preference: the seam our fused op exposes +(`cos_sin` as a per-token cache, instead of a cache plus positions) is strictly +more general than upstream's, and the divergence is upstream's limitation rather +than ours. If upstream lands MRoPE support in `fused_qk_norm_rope.py` and drops +the conjunct, this exception is resolved by upstream converging on us, and the +record is then deleted rather than migrated. + +**What the exception does NOT excuse — and what this wave actually repairs.** +An exception on the KERNEL gate says nothing about the BENCHMARK gate. #414's +defect is a denominator, not a kernel. At `multimodal.py:78` the flag defaults +`False`, so a Qwen3.6 checkpoint that loads as +`Qwen3_5*ForConditionalGeneration` gives the oracle `text_only == False` and +four ops per full-attention layer, while our arm has issued one launch by +default throughout. Comparing those compares two algorithms, which AGENTS.md +§Gates forbids, and it flatters us on exactly the TTFT axis where our deficit +sits. + +The correct denominator is vLLM **with** `--language-model-only`, on two +independent grounds that agree. It is a production configuration: 43 of the 157 +official `recipes.vllm.ai` recipes pass it. And AGENTS.md §Gates requires +vLLM's production configuration rather than a handicapped one — the rule that +forbids `--enforce-eager` as a denominator forbids this for the same reason. +The 2026-08-13 clock-controlled series already adopted it and is the binding +record. + +**The residual, which is what L4 closes.** That series did not run through the +canonical driver. `scripts/dgx-online-serving.sh::start_server` — the driver the +superseded grids used, and the one the next campaign reaches for — still +launched the oracle with no `--language-model-only`, while +`tools/bench/run_serve_low.py` passed it. Two harnesses disagreeing about the +oracle's configuration is #414 waiting to recur, and a grep for the flag reads +as coverage while the canonical path lacks it. L4 makes them agree and makes the +agreement a gate: `scripts/check-oracle-denominator-flags.py` refuses any +harness that launches the pinned oracle server against a multimodal-architecture +gate checkpoint without the flag. + +**Risks.** + +1. *The exception decays into a licence.* Mitigated by scope: it covers exactly + the `text_only` conjunct of one predicate, and it is argued from a seam + difference a reader can check in one function signature. +2. *Our eager arm is silently wrong on MRoPE rather than refusing.* + `VT_FUSE_ATTN_PREAMBLE=0` is a supported same-binary A/B rollback; combined + with an MRoPE cache it produces 1-D RoPE and wrong tokens with no diagnostic. + Found by this wave, filed as its own bug, and listed under `## Owed` — its + gate is a VL token-exactness run and needs a GPU, which this wave did not + have. +3. *The checker becomes a lock.* It reads harness sources and holds its + expectation in its own code, so no pull request has to edit a shared record + to pass it. +4. *A harness that legitimately launches no oracle trips it.* The checker keys + on the oracle `serve` invocation, not on the file, so such a harness is + simply out of its scope. +5. *The detector degrades into a list of spellings.* Raised by the fresh review + of this wave. `_ORACLE_CLIENT` first named the two variables this tree + happens to use, `${client}` and `${VLLM_ORACLE}`, so a future harness writing + `"$ORACLE" serve` or `"$server_bin" serve` would have gone unscanned — and an + unscanned launch reads as a clean tree, which is the failure the exit-2 + "broken detector" arm exists to catch and could not have caught here, because + the other launches keep the count above zero. The pattern now matches the + binary spelled out OR any `$`-expansion, since what identifies a launch is + `serve` sitting directly after the binary, not the name of the variable + holding it. `bench serve` is unaffected: the token before `serve` there is + `bench`, which is no expansion. The real tree discovers the same three + launches, so the widening added no false positive. + +**Tests.** + +- `tests/scripts/test_check_oracle_denominator_flags.py` — the checker over + synthetic harness fixtures: an oracle `serve` line without the flag is + REFUSED; the same line with the flag PASSES; a non-oracle command is ignored; + a text-only-architecture harness is out of scope; and the two real in-tree + harnesses are asserted clean. RED-first is behavioural — the checker run + against `dgx-online-serving.sh` as it stood before this change exits non-zero + naming that file and that line. A second RED-first pair covers risk 5: + `test_any_variable_spelling_of_the_client_is_still_a_launch` failed 4/4 of its + spellings before the pattern widened, each reporting exit 2 and "found NO + oracle server launch at all", and + `test_bench_serve_stays_a_client_under_any_variable_spelling` holds the + widening off the timed client. +- No upstream test covers this. `fused_qk_norm_rope.py` has no test module at + the pin, and upstream has no test asserting that a benchmark harness + configures its own oracle, because the harness is ours. Searched at the pin: + `tests/models/**`, `tests/benchmarks/**`, `tests/config/**`, + `tests/multimodal/**`, `tests/kernels/**`. Recorded as a search result, not as + an absence claim. + +**Gates.** `scripts/check-oracle-denominator-flags.py` exit 0; +`tests/scripts/test_check_oracle_denominator_flags.py` green; the full +`scripts/agent-preflight.sh` gate; `check-commit-trailers.py` and +`check-commit-style.py` over the range. No CUDA gate is claimed and none is +needed: this wave changes no product code path. + +**Evidence.** Every upstream `file:line` in the table above was read in the +pinned checkout at `555967922`. The seam asymmetry is readable in one place — +`vt::AttnQkNormRopeGate` takes `cos_sin` and no positions, where +`fused_qk_rmsnorm_rope_gate` takes `cos_sin_cache` **and** `positions`. + +**What this invalidates, exactly.** The first draft of this section said +"nothing currently binding", on the reasoning that the 27B and 35B figures #414 +flattered are already recorded as SUPERSEDED and OPTIMISTIC in +`docs/BENCHMARKS.md` and in the benchmark record's "What this supersedes, and in +which direction" table, and that the binding rows for both gate models already +carry `--language-model-only`. That much is true and unchanged. The conclusion +drawn from it was wrong, because it only looked at the two gate models. + +**`docs/BENCHMARKS.md` Qwen3.5-4B `1.0283x tput` is affected and was NOT marked.** +The chain, each link checked rather than assumed: + +1. The row cites [`bench-evidence/qwen35-4b-sm120-main-20260807.md`](../../docs/bench-evidence/qwen35-4b-sm120-main-20260807.md), + whose reproduction identity names `Qwen/Qwen3.5-4B` and the exact workload + `run_qwen35_4b_compare.sh` drives. +2. `tools/bench/run_qwen35_4b_compare.sh:120` runs the vLLM arm through + `tools/bench/vllm_closed_loop_metrics.py`, whose `LLM(...)` at `:160` never + passes `language_model_only` and exposes no way to set it — the #1345 surface. +3. Upstream registers `Qwen/Qwen3.5-4B` under + **`Qwen3_5ForConditionalGeneration`** (`tests/models/registry.py:1322-1324`, + `extras={"4b": "Qwen/Qwen3.5-4B"}`), so its `multimodal_config` is non-`None` + and `text_only` is `False`. +4. The conjunct was live at that measurement. `git log -S'use_fused_qk_norm_rope_gate'` + over `qwen3_next.py` dates its introduction to `16282a9c4`, 2026-06-10, two + months before the 2026-08-07 run. +5. The remaining conjuncts hold: sm_120 is CUDA, and Qwen3.5-4B carries + `attn_output_gate` with NeoX-style RoPE. + +So that leg's oracle ran the UNFUSED preamble on its full-attention layers while +our arm ran the fused one. **Direction: it flatters us**, for the same reason +#414 gives. **Magnitude: unmeasured, and not estimated here** — the model is +GDN-hybrid, so only its full-attention layers are exposed, and this wave took no +measurement. The row is marked rather than withdrawn, per AGENTS.md: evidence is +annotated, never deleted. Re-measurement is owed with #1345. + +**Not affected, and why, so the next reader does not re-derive it.** #414 reaches +a figure only where the full-attention layers are `Qwen3NextAttention` AND the +checkpoint loads as a `*ForConditionalGeneration`. That is the Qwen3.5/3.6/3.8 +family alone. The OPT, GLM-4, InternLM2, Qwen3-dense, Qwen3-Coder and +DeepSeek-V2-Lite legs do not construct `Qwen3NextAttention` at all. The +`Qwen3.8-27B` rows ran through `tools/bench/run_serve_low.py`, which has passed +`--language-model-only` since it was written. The Qwen3.5-4B GDN prefill +kernel row is conv and post-conv timing on the LINEAR-attention path, which the +full-attention preamble does not touch. + +This wave therefore withdraws no number, quotes no new one, marks one, and +removes the mechanism by which the next canonical run would have produced +another flattered set. + +**Stop conditions.** Stop and escalate if upstream drops the `text_only` +conjunct (the exception is then obsolete, not merely stale); if our fused +preamble ever stops being the MRoPE arm (the exception's ground disappears with +it); or if a harness needs the oracle launched WITHOUT the flag for a reason +other than a text-only architecture — that is a denominator decision for the +operator, not a checker allowlist entry. + +**What L4 does NOT do, so the next wave knows what it inherits.** It changes no +product code path, so it claims no token gate and no measurement. It gates the +CLI oracle-launch surface only; the in-process `LLM(...)` surface carries the +same defect and is owed below, deliberately, because those harnesses take +`--model` as a path and no static rule can know whether a run points at a +multimodal checkpoint. And it takes no measurement at all: the flag is now +correct in the canonical driver, but no grid has yet been run through the +repaired driver, so nothing here supersedes or replaces a published number. + --- ## 2. Structured contract @@ -992,3 +1226,19 @@ pieces).** near-term gate. - No HW-blocked modality for the Qwen3.6 target: the vision tower is ~1 GiB and fits the 119 GiB unified pool alongside the 27B/35B LLM. + +--- + +## Owed + +Carried by `ENG-MM-INPUT-PIPELINE`, filed while landing L4 (§1.6). + +- [#1340](https://github.com/mudler/vllm.cpp/issues/1340) — `VT_FUSE_ATTN_PREAMBLE=0` + on the MRoPE path silently applies 1-D RoPE instead of refusing. Needs a GPU + VL token-exactness run through `ModelRegistry::Forward` to gate. +- [#1345](https://github.com/mudler/vllm.cpp/issues/1345) — the three in-process + bench harnesses (`profile_vllm_online_gate.py:209`, + `vllm_closed_loop_metrics.py:160`, `dump_vllm_tokens.py:34`) construct the + oracle with `language_model_only` at its `False` default and expose no way to + set it. Needs the knob threaded through and the resolved value recorded beside + the measurement; the default is a denominator decision for the operator. diff --git a/.agents/specs/sm120-qwen35-pareto-2026-08-09.md b/.agents/specs/sm120-qwen35-pareto-2026-08-09.md index 509875a17..5697cf974 100644 --- a/.agents/specs/sm120-qwen35-pareto-2026-08-09.md +++ b/.agents/specs/sm120-qwen35-pareto-2026-08-09.md @@ -328,7 +328,8 @@ file moved and the row stays `GATING`. What git shows: - The exact 4B series has been **rerun six times since** — `qwen35-4b-*` `20260727`, `20260728`, `20260729`, `20260803`, `20260805`, `20260807` in `docs/bench-evidence/` — and the throughput axis moved from **0.9864x** to - **1.0283x PASS** (6831.71 vs 6643.40 tok/s, three repetitions). + **1.0283x PASS** (6831.71 vs 6643.40 tok/s, three repetitions) — OPTIMISTIC, + because that denominator ran UNFUSED ([#1345](https://github.com/mudler/vllm.cpp/issues/1345), #414). - The successor lever on the same wait was tried and **rejected**: geometric argmax scratch migrated the wait rather than removing it and cost +1% TPOT ([spec](sm120-qwen35-argmax-scratch-growth-2026-08-09.md)). diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 339af87c2..c1f0cad49 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -262,6 +262,15 @@ jobs: run: | python3 scripts/check-surface-coverage.py python3 tests/scripts/test_check_surface_coverage.py + - name: No harness launches the pinned oracle on a handicapped path (#414) + # Without `--language-model-only` a Qwen3.6 checkpoint gives the oracle + # `text_only == False` (qwen3_next.py:324-331 at the pin), so it runs its + # UNFUSED QK-norm+RoPE+gate path against our fused one and every ratio it + # produces is FLATTERED. The 2026-08-13 series repaired the RUN; this + # repairs the HARNESS, which is what the next campaign reaches for. + run: | + python3 scripts/check-oracle-denominator-flags.py + python3 tests/scripts/test_check_oracle_denominator_flags.py - name: The x86_64 CPU floor harness runs from a clean checkout # It shipped unable to run at all: `OUT=evi` with no `mkdir -p`, so every # redirection failed, every leg was discarded for a non-zero exit, and diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index 183716c4a..19d1e01e9 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -58,7 +58,7 @@ The first series free of both, at the pin, graphed, and at a pinned clock is in | Qwen3.6-27B | NVFP4 (`nvidia` @`0893e160`, ModelOpt `modelopt_mixed`) | 0.25.0 ROLLBACK, SUPERSEDED | 0/6 | BEHIND, uniformly 0.94x on decode, flat c1-c32, but **VOID as a ratio** (#520, #414). At the pin, clocks pinned: **0.976x c1 / 0.946x c4 TPOT**. Different model from the `unsloth` row (NVFP4 MLP + FP8 W8A8 tower) | | Qwen3.6-35B-A3B | NVFP4 `modelopt_mixed` | 0.25.0 ROLLBACK, SUPERSEDED | 2/18 | 3-rep grid 2026-08-05 @`1ea26427`: 0.93-1.03x, VOID as ratios (#520, #414). At the pin, clocks pinned: **0.995x c1 / 0.946x c4 TPOT**. ★ probe found a prod async batch-1 greedy DEGENERATION bug the mirror fixes | | DeepSeek-V2-Lite | bf16 MLA | 0.25.0 ROLLBACK, SUPERSEDED | 4/25 | Attributed miss, row stays `ACTIVE` | -| Qwen3.5-4B | bf16 direct-load | 0.26.0.dev0 | **1.0283x tput, `PENDING`** | OPEN: TTFT/TPOT/E2E 1.085/1.017/1.029x, VRAM +118.7 MiB ([data](bench-evidence/qwen35-4b-sm120-main-20260807.md)) | +| Qwen3.5-4B | bf16 direct-load | 0.26.0.dev0, UNFUSED and vision-tower-carrying denominator ([#1345](https://github.com/mudler/vllm.cpp/issues/1345)) | **1.0283x tput, `PENDING` and OPTIMISTIC** | OPTIMISTIC on all axes (#414). TTFT/TPOT/E2E 1.085/1.017/1.029x; VRAM **+118.7 MiB** OPEN, not like-for-like: the oracle built the tower the flag elides ([data](bench-evidence/qwen35-4b-sm120-main-20260807.md)) | | Qwen3.8-27B | bf16 (@`1d4bf0f2`) | 0.26.0.dev0 at the pin, graphed; c4 at a pinned 2184 MHz, the 2026-08-19 re-measure SAMPLED only | **1 of 3 concurrency cells** | Token gate PASSES. c4 like-for-like: tput **0.963x**, ITL **1.008x**. c1 re-measured COMPLETE, pairing DISCARDED on clock spread; c8 vLLM denominator NOT MEASURABLE ([#915](https://github.com/mudler/vllm.cpp/issues/915)) | ### GDN prefill kernels by GPU diff --git a/scripts/agent-preflight.sh b/scripts/agent-preflight.sh index 6ea2a84b3..a094cf60e 100755 --- a/scripts/agent-preflight.sh +++ b/scripts/agent-preflight.sh @@ -106,6 +106,7 @@ CHECKERS=( check-test-registration check-snapshot-pins check-oracle-pins + check-oracle-denominator-flags check-now-current check-gate-commands check-symbol-anchors @@ -163,6 +164,7 @@ SUITES=( test_agent_preflight_skip_report test_agent_pr_body test_check_symbol_anchors + test_check_oracle_denominator_flags test_check_conflict_markers ) diff --git a/scripts/check-oracle-denominator-flags.py b/scripts/check-oracle-denominator-flags.py new file mode 100755 index 000000000..927017456 --- /dev/null +++ b/scripts/check-oracle-denominator-flags.py @@ -0,0 +1,254 @@ +#!/usr/bin/env python3 +"""Refuse a harness that launches the pinned vLLM oracle SERVER on a handicapped path. + +This is the structural half of +[#414](https://github.com/mudler/vllm.cpp/issues/414), and the gate that #607 +wave L4 owes (`.agents/specs/multimodal-track.md` §1.6). + +The defect it removes +--------------------- +At the pin, `vllm/config/multimodal.py:78` defaults `language_model_only` to +`False`. A Qwen3.6 checkpoint loads as `Qwen3_5*ForConditionalGeneration`, so its +`multimodal_config` is non-`None`, so +`vllm/model_executor/models/qwen3_next.py:324-331` computes `text_only == False` +and DISABLES `use_fused_qk_norm_rope_gate`. The oracle then issues four ops per +full-attention layer where its own production configuration issues one, while +our arm has issued the single fused launch by default throughout +(`src/vllm/model_executor/models/qwen3_5.cpp::FuseAttnPreambleOn`). + +A ratio measured that way compares two different algorithms, which AGENTS.md +§Gates forbids, and the error runs in OUR favour, hardest on TTFT. The +2026-08-13 clock-controlled series repaired it by passing +`--language-model-only`. That repaired the RUN, not the HARNESS: +`scripts/dgx-online-serving.sh` still launched the oracle without the flag while +`tools/bench/run_serve_low.py` passed it, so the two disagreed about the +oracle's configuration and the next canonical campaign would have reproduced the +defect. A grep for the flag reads as coverage while the canonical driver lacks +it, which is why this is a checker and not a comment. + +Scope, and what is deliberately outside it +------------------------------------------ +This gate covers the ONE surface where the published parity ratios come from: a +command that launches the oracle SERVER through its CLI. A token `serve` +immediately following an oracle client token (`vllm`, `.../vllm`, or a +`${client}`-style variable) is a launch. `vllm bench serve` is the timed CLIENT +rather than a server, and is not matched, because `serve` there follows `bench`. + +The same defect exists on the IN-PROCESS surface: `LLM(...)` in +`tools/bench/{profile_vllm_online_gate,vllm_closed_loop_metrics,dump_vllm_tokens}.py` +constructs the oracle with `language_model_only` left at its `False` default and +exposes no way to set it. Those harnesses take `--model` as a path, so no static +rule can know whether a given run points at a multimodal checkpoint; the repair +is to thread the knob through and record the resolved value beside the +measurement. That is tracked separately and is NOT enforced here, because a +checker that cannot decide its own question is worse than none. + +Each matched launch must pass `--language-model-only` or carry an explicit +exemption naming a reason: + + # ORACLE-DENOMINATOR-EXEMPT: + +An exemption is for an architecture with no `multimodal_config` at all, where +`text_only` is already `True` and the flag decides nothing. It is not a place to +park a multimodal checkpoint. + +Why this is not a lock +---------------------- +The checker holds its expectation in its own code and reads harness sources. No +pull request has to edit a shared record to pass it, and adding a harness costs +nothing unless that harness launches an oracle server. + + scripts/check-oracle-denominator-flags.py # gate + scripts/check-oracle-denominator-flags.py --json # the discovered set + scripts/check-oracle-denominator-flags.py --root DIR # scan another tree + +`--json` exists so a test can pin the DISCOVERED set, not only the violation +count. A detector whose terms stopped matching would otherwise report a clean +tree, which is the failure this project has already paid for elsewhere: a null +grep proves the terms wrong, never an absence. +""" + +from __future__ import annotations + +import argparse +import json +import re +import sys +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[1] + +# Harness surfaces. A file outside these globs cannot launch a gate oracle +# without first becoming a harness, at which point it lands here in the same +# change. +SCAN_GLOBS = ("scripts/*.sh", "scripts/*.py", "tools/bench/*.py", "tools/bench/*.sh") + +# This checker reads itself out of the scan: its own prose names every token it +# looks for, and matching that prose would be a tautology. +SELF = "check-oracle-denominator-flags.py" + +REQUIRED_FLAG = "--language-model-only" +EXEMPT_MARKER = "ORACLE-DENOMINATOR-EXEMPT:" + +# A token naming the pinned oracle CLI: either the binary spelled out, or ANY +# shell or Python variable holding it. Naming the two variables this tree +# happens to use today would have made the detector a list of spellings, and a +# harness calling its binary `${ORACLE}` or `${server_bin}` would have launched +# the oracle unscanned while the checker reported a clean tree. What identifies +# a launch is `serve` sitting directly after the binary, so the binary may be +# anonymous. `bench serve` is unaffected and stays a client: the token before +# `serve` there is `bench`, which is no expansion at all. +# +# Case-sensitive on purpose: prose writes "vLLM", commands write "vllm". The +# optional closing brace is not cosmetic -- `_TOKEN_STRIP` eats `}` off the end +# of a token, so `"${client}"` reaches this pattern as `${client`. +_ORACLE_CLIENT = re.compile(r"\A(?:(?:.*/)?vllm|\$\{?[A-Za-z0-9_]+\}?)\Z") +_TOKEN_STRIP = "\"'`,()[]{}\\" +# A line that opens a command block, so a flag placed before `serve` still counts. +_OPENER = re.compile(r"[(\[]\s*$") +# A line that closes one. +_CLOSER = re.compile(r"\A\s*[)\]]\s*,?\s*\Z") + + +def _tokens(line: str) -> list[str]: + return [t.strip(_TOKEN_STRIP) for t in line.split()] + + +def _is_comment(line: str) -> bool: + stripped = line.lstrip() + return stripped.startswith("#") or stripped.startswith("*") + + +def _block(lines: list[str], index: int) -> tuple[int, int]: + """The command block containing line ``index`` (0-based), as [start, end).""" + start = index + while start > 0: + if _OPENER.search(lines[start - 1]): + start -= 1 + break + if not lines[start - 1].strip(): + break + start -= 1 + end = index + 1 + while end < len(lines): + if _CLOSER.match(lines[end]): + end += 1 + break + if not lines[end].strip(): + break + end += 1 + return start, end + + +def _exempt(lines: list[str], start: int, end: int) -> str | None: + """The exemption reason for a block, searching the block and its 8-line lead.""" + lead = max(0, start - 8) + for line in lines[lead:end]: + if EXEMPT_MARKER in line: + reason = line.split(EXEMPT_MARKER, 1)[1].strip().strip(_TOKEN_STRIP).strip() + return reason or None + return None + + +def _serve_launches(lines: list[str]) -> list[int]: + """0-based indices of lines launching the oracle SERVER through its CLI.""" + found = [] + for i, line in enumerate(lines): + if _is_comment(line): + continue + tokens = _tokens(line) + for j, token in enumerate(tokens): + if token != "serve" or j == 0: + continue + if _ORACLE_CLIENT.match(tokens[j - 1]): + found.append(i) + break + return found + + +def scan(root: Path) -> list[dict]: + """Every oracle server launch in the tree, with its verdict.""" + results: list[dict] = [] + paths: list[Path] = [] + for glob in SCAN_GLOBS: + paths.extend(sorted(root.glob(glob))) + for path in paths: + if path.name == SELF: + continue + try: + lines = path.read_text(encoding="utf-8", errors="replace").splitlines() + except OSError: + continue + for index in _serve_launches(lines): + start, end = _block(lines, index) + body = "\n".join(lines[start:end]) + results.append( + { + "file": str(path.relative_to(root)), + "line": index + 1, + "has_flag": REQUIRED_FLAG in body, + "exempt_reason": _exempt(lines, start, end), + } + ) + results.sort(key=lambda r: (r["file"], r["line"])) + return results + + +def violations(results: list[dict]) -> list[dict]: + return [r for r in results if not r["has_flag"] and not r["exempt_reason"]] + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + parser.add_argument("--root", type=Path, default=ROOT) + parser.add_argument("--json", action="store_true", help="print the discovered set") + args = parser.parse_args(argv) + + results = scan(args.root) + if args.json: + print(json.dumps(results, indent=2, sort_keys=True)) + + if not results: + print( + "check-oracle-denominator-flags: found NO oracle server launch at all. " + "That is a broken detector, not a clean tree -- this repository has at " + "least one harness that launches the pinned oracle.", + file=sys.stderr, + ) + return 2 + + bad = violations(results) + if bad: + for r in bad: + print( + f"{r['file']}:{r['line']}: launches the pinned vLLM oracle server " + f"without {REQUIRED_FLAG}. The oracle then runs its UNFUSED " + f"QK-norm+RoPE+gate path (qwen3_next.py:324-331 at the pin) while " + f"our arm runs the fused one, so every ratio from this harness is " + f"FLATTERED (#414). Pass the flag, or mark the block " + f"'{EXEMPT_MARKER} ' when the architecture has no " + f"multimodal_config and the flag decides nothing.", + file=sys.stderr, + ) + print( + f"\n{len(bad)} handicapped oracle launch(es) of {len(results)} " + f"discovered. Never weaken this checker to make a harness pass: the " + f"denominator it protects is the whole comparison.", + file=sys.stderr, + ) + return 1 + + exempt = [r for r in results if r["exempt_reason"]] + print( + f"check-oracle-denominator-flags: {len(results)} oracle server launch(es), " + f"{len(results) - len(exempt)} carrying {REQUIRED_FLAG}, " + f"{len(exempt)} exempt." + ) + for r in exempt: + print(f" exempt {r['file']}:{r['line']} -- {r['exempt_reason']}") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/dgx-online-serving.sh b/scripts/dgx-online-serving.sh index 82fdf6284..98c8dc226 100755 --- a/scripts/dgx-online-serving.sh +++ b/scripts/dgx-online-serving.sh @@ -481,6 +481,9 @@ start_server() { # backend aborts engine start (BackendSupportedError mm_fp4 cap 121), so we # force the Marlin W4A16 keep-quant kernel our arm mirrors. Dense Qwen3 has no # mamba SSM state, so the hybrid-only SSM cache-dtype flag is omitted here. + # ORACLE-DENOMINATOR-EXEMPT: dense Qwen3-8B has no multimodal_config, so text_only is already True + # and the flag decides nothing. multimodal.py:78 never applies, qwen3_next.py:325 already resolves + # True, and adding the flag to a measured harness that does not need it is noise, not rigour. server_cmd=( env "PATH=$(dirname "${client}"):${PATH}" "VLLM_DISABLED_KERNELS=FlashInferMxFp4LinearKernel" @@ -493,10 +496,24 @@ start_server() { --port "${port}" ) else + # 27 / 27n / 35 are Qwen3.6 checkpoints, which load as + # Qwen3_5*ForConditionalGeneration. Their multimodal_config is therefore + # non-None, and multimodal.py:78 defaults language_model_only to False, so + # without the flag qwen3_next.py:325 computes text_only == False and the + # oracle DISABLES use_fused_qk_norm_rope_gate: four ops per full-attention + # layer where its own production configuration issues one, against our arm's + # single fused launch. That is a handicapped denominator whose cost scales + # with prompt tokens, so it lands hardest on TTFT, and it flatters us (#414). + # `tools/bench/run_serve_low.py` has passed the flag since it was written; + # this driver did not, so the two harnesses disagreed about the oracle's own + # configuration. The 2026-08-13 clock-controlled series passed it and is the + # binding record; this makes the canonical driver agree with it, and + # scripts/check-oracle-denominator-flags.py keeps them agreeing. server_cmd=( env "PATH=$(dirname "${client}"):${PATH}" "${client}" serve "${snapshot}" --served-model-name gate + --language-model-only --gpu-memory-utilization 0.6 --max-num-seqs "${max_num_seqs}" --max-num-batched-tokens "${max_num_batched_tokens}" diff --git a/tests/scripts/test_check_oracle_denominator_flags.py b/tests/scripts/test_check_oracle_denominator_flags.py new file mode 100644 index 000000000..38a5b51fa --- /dev/null +++ b/tests/scripts/test_check_oracle_denominator_flags.py @@ -0,0 +1,221 @@ +#!/usr/bin/env python3 +"""Unit and mutation checks for scripts/check-oracle-denominator-flags.py (#414, #607). + +The case this suite exists for is `test_real_tree_discovers_the_known_launches`. +A detector that stopped matching would report a clean tree, and a clean tree is +what a broken detector and a correct one both look like from the exit status +alone. So the discovered SET is pinned by file, not only the violation count, +and the checker itself exits 2 rather than 0 when it finds nothing at all. + +`test_flag_removed_from_the_canonical_driver_goes_red` is the mutation: it is +the state the tree was actually in before #607 wave L4, and it must be +detectable, not merely described. +""" + +from __future__ import annotations + +import json +import subprocess +import sys +import tempfile +import unittest +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[2] +CHECKER = ROOT / "scripts/check-oracle-denominator-flags.py" +DRIVER = ROOT / "scripts/dgx-online-serving.sh" + + +def run(*args: str) -> subprocess.CompletedProcess: + return subprocess.run( + [sys.executable, str(CHECKER), *args], + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, + text=True, + check=False, + ) + + +def tree(**files: str) -> tempfile.TemporaryDirectory: + """A scratch root carrying the given harness files.""" + tmp = tempfile.TemporaryDirectory() + root = Path(tmp.name) + for relative, body in files.items(): + path = root / relative + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(body, encoding="utf-8") + return tmp + + +# The real shape, reduced: an oracle server launched from a shell array. +SERVE_NO_FLAG = """\ +start_server() { + server_cmd=( + env "PATH=$(dirname "${client}"):${PATH}" + "${client}" serve "${snapshot}" + --served-model-name gate + --port "${port}" + ) +} +""" + +SERVE_WITH_FLAG = SERVE_NO_FLAG.replace( + " --served-model-name gate\n", + " --served-model-name gate\n --language-model-only\n", +) + +SERVE_EXEMPT = SERVE_NO_FLAG.replace( + " server_cmd=(\n", + " # ORACLE-DENOMINATOR-EXEMPT: dense text arch, no multimodal_config\n" + " server_cmd=(\n", +) + +SERVE_EXEMPT_NO_REASON = SERVE_NO_FLAG.replace( + " server_cmd=(\n", + " # ORACLE-DENOMINATOR-EXEMPT:\n server_cmd=(\n", +) + + +class OracleDenominatorFlagsTest(unittest.TestCase): + def test_a_serve_launch_without_the_flag_is_refused(self) -> None: + with tree(**{"scripts/h.sh": SERVE_NO_FLAG}) as root: + got = run("--root", root) + self.assertEqual(got.returncode, 1, got.stdout) + self.assertIn("scripts/h.sh:4", got.stdout) + self.assertIn("--language-model-only", got.stdout) + self.assertIn("#414", got.stdout) + + def test_a_serve_launch_with_the_flag_passes(self) -> None: + with tree(**{"scripts/h.sh": SERVE_WITH_FLAG}) as root: + got = run("--root", root) + self.assertEqual(got.returncode, 0, got.stdout) + self.assertIn("1 oracle server launch(es)", got.stdout) + + def test_an_exemption_with_a_reason_passes_and_prints_the_reason(self) -> None: + with tree(**{"scripts/h.sh": SERVE_EXEMPT}) as root: + got = run("--root", root) + self.assertEqual(got.returncode, 0, got.stdout) + self.assertIn("dense text arch, no multimodal_config", got.stdout) + + def test_an_exemption_without_a_reason_is_not_an_exemption(self) -> None: + with tree(**{"scripts/h.sh": SERVE_EXEMPT_NO_REASON}) as root: + got = run("--root", root) + self.assertEqual(got.returncode, 1, got.stdout) + + def test_bench_serve_is_the_timed_client_and_is_not_a_launch(self) -> None: + # `vllm bench serve` issues the requests. Matching it would demand a + # server flag on a client, which vLLM would reject. + body = 'cmd=(\n "${client}" bench serve --backend openai\n)\n' + with tree(**{"scripts/h.sh": body}) as root: + got = run("--root", root) + self.assertEqual(got.returncode, 2, got.stdout) + self.assertIn("broken detector", got.stdout) + + def test_any_variable_spelling_of_the_client_is_still_a_launch(self) -> None: + # The evasion surface. The detector used to name the two variables this + # tree happens to use, `${client}` and `${VLLM_ORACLE}`, so a harness + # that called its binary anything else launched the oracle unscanned and + # the checker reported a clean tree. What identifies a launch is the + # `serve` subcommand sitting directly after the binary, not the name + # somebody gave the variable holding it. + for spelling in ('"$ORACLE"', '"${server_bin}"', "$oracle_cli", "${VENV}"): + with self.subTest(spelling=spelling): + body = f'cmd=(\n {spelling} serve "${{snapshot}}" --port 8000\n)\n' + with tree(**{"scripts/h.sh": body}) as root: + got = run("--root", root) + self.assertEqual(got.returncode, 1, got.stdout) + self.assertIn("scripts/h.sh:2", got.stdout) + + def test_bench_serve_stays_a_client_under_any_variable_spelling(self) -> None: + # The guard on the line above. Widening to any `$`-expansion must not + # start matching `bench serve`, which is the timed CLIENT: the token + # before `serve` there is `bench`, which is no expansion at all. + for spelling in ('"$ORACLE"', '"${server_bin}"', '"${client}"'): + with self.subTest(spelling=spelling): + body = f'cmd=(\n {spelling} bench serve --backend openai\n)\n' + with tree(**{"scripts/h.sh": body}) as root: + got = run("--root", root) + self.assertEqual(got.returncode, 2, got.stdout) + self.assertIn("broken detector", got.stdout) + + def test_prose_naming_the_flag_free_oracle_is_not_a_launch(self) -> None: + # Case matters: prose writes "vLLM", commands write "vllm". A comment + # line is never a command either. + body = ( + '# the vLLM serve arm runs without it\n' + 'echo "see vllm/benchmarks/serve.py for the client"\n' + 'true_cmd=(\n "${client}" serve "${snap}" --language-model-only\n)\n' + ) + with tree(**{"scripts/h.sh": body}) as root: + got = run("--root", root, "--json") + self.assertEqual(got.returncode, 0, got.stdout) + payload = json.loads(got.stdout[got.stdout.index("[") : got.stdout.rindex("]") + 1]) + self.assertEqual([r["line"] for r in payload], [4], payload) + + def test_a_python_harness_launch_is_matched_too(self) -> None: + body = ( + '"vllm_server": [\n' + ' "/bin/vllm", "serve", "",\n' + ' "--served-model-name", "gate",\n' + "],\n" + ) + with tree(**{"tools/bench/h.py": body}) as root: + got = run("--root", root) + self.assertEqual(got.returncode, 1, got.stdout) + self.assertIn("tools/bench/h.py:2", got.stdout) + + def test_an_empty_tree_is_a_broken_detector_not_a_pass(self) -> None: + with tree(**{"scripts/h.sh": "echo nothing here\n"}) as root: + got = run("--root", root) + self.assertEqual(got.returncode, 2, got.stdout) + + def test_real_tree_is_clean(self) -> None: + got = run() + self.assertEqual(got.returncode, 0, got.stdout) + + def test_real_tree_discovers_the_known_launches(self) -> None: + got = run("--json") + self.assertEqual(got.returncode, 0, got.stdout) + payload = json.loads(got.stdout[got.stdout.index("[") : got.stdout.rindex("]") + 1]) + files = sorted({r["file"] for r in payload}) + self.assertEqual( + files, + ["scripts/dgx-online-serving.sh", "tools/bench/run_serve_low.py"], + payload, + ) + # Both arms of the canonical driver, plus the low-batch harness. A drop + # below three means the detector stopped seeing a launch it used to see. + self.assertGreaterEqual(len(payload), 3, payload) + self.assertTrue( + any(r["exempt_reason"] for r in payload), + "the q3mxfp4 dense arm should be the exempt one", + ) + self.assertTrue( + any( + r["file"] == "tools/bench/run_serve_low.py" and r["has_flag"] + for r in payload + ), + payload, + ) + + def test_flag_removed_from_the_canonical_driver_goes_red(self) -> None: + # The pre-L4 state of the tree, reconstructed. This is the mutation that + # proves the gate detects the defect rather than describing it. + original = DRIVER.read_text(encoding="utf-8") + self.assertIn(" --language-model-only\n", original) + mutated = original.replace(" --language-model-only\n", "", 1) + self.assertNotEqual(mutated, original, "mutation did not apply") + with tempfile.TemporaryDirectory() as name: + root = Path(name) + (root / "scripts").mkdir() + (root / "scripts" / DRIVER.name).write_text(mutated, encoding="utf-8") + got = run("--root", str(root)) + self.assertEqual(got.returncode, 1, got.stdout) + self.assertIn("dgx-online-serving.sh", got.stdout) + # The exempt q3mxfp4 arm must stay exempt: the mutation removes one + # launch's flag, not the exemption of the other. + self.assertEqual(got.stdout.count("launches the pinned vLLM oracle"), 1, got.stdout) + + +if __name__ == "__main__": + unittest.main(verbosity=2)