diff --git a/.agents/issue-index.md b/.agents/issue-index.md index 30c39e4ba..7d575911e 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -668,3 +668,4 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#1804](https://github.com/mudler/vllm.cpp/issues/1804) | `MODEL-MM-dots3-note-dots3-note-for-causal-lm` | **`.agents/specs/dots3-note.md` §4 item 6 read `is_neox_style=False` as belonging to the dots3-note SLIDING rope ONLY, and it belongs to both MLA ropes** — the sentence would have sent a W3 implementer to rotate the 13 full-attention layers split-half NeoX. `Dots3NoteSlidingAttention` does pass `is_neox_style=False` literally (`model.py:408` @ vLLM `origin/main` `c205726108df54bb6fbf15b19e725a4a3add2b18`), which is the half W0 read; `Dots3NoteFullAttention` (`model.py:219`) inherits the SAME hard-coded value from `deepseek_v2.py`::`DeepseekV2MLAAttention.__init__` (`:1093-1098`). So the two geometries do NOT differ on the RoPE layout at all — they differ on the THETA, `swa_rope_theta` 5e4 on 33 layers against `rope_theta` 8e7 on 13. The polarity that DOES flip is the INDEXER's, and that is §4 trap 2's point rather than item 6's: `deepseek_v2.py:1148` sets the indexer rope to `is_neox_style = not indexer_rope_interleave`, so at DeepSeek-V3.2's absent-key default the indexer runs NeoX beside an MLA rope that is GPT-J, and `indexer_rope_interleave = True` (`configs/dots3_note.py:23`) is what makes dots3-note's two agree. Being wrong in this direction is SILENT: the same 64 coordinates are rotated either way, so nothing changes shape and nothing throws — the §4 defect class exactly, on a row that spec §6.4 says has no oracle anywhere to catch it. FIXED IN FLOW on `row/MODEL-MM-dots3-note-W1`: §4 item 6 corrected in place and it says what it used to say (`main` is never rewritten), `ParseDots3NoteParams` resolves `rope_is_neox_style = false` on BOTH geometries with the two citations beside it, and `tests/vllm/models/test_dots3_note_scaffold.cpp` asserts both plus the indexer's agreement with them. The assertion was captured RED against the NeoX reading, on an arm that compiled and ran, before the corrected value existed | bug | | [#1802](https://github.com/mudler/vllm.cpp/issues/1802) | `BACKEND-CUDA-SM110` | **Re-measuring the sm_110 `ctest` baseline at `6756f9131` (2026-08-23, job `8bf39567-9334-4f7e-aa27-43a2aa867bb7`, artifacts `/mnt/nas_share/rc/thor-w05-955/out/`) found six names red that were green at `0764ded2b`, none with an owner**, plus two standing entries whose recorded cause no longer holds. Arrivals, all mode `Failed`: `test_cuda_ops` (`:106` `CHECK(bad == 0)` -> `6 == 0` and `7 == 0`, silu_and_mul vs CPU; also red on GB10 at 439/440 against 438/440 here, [#907](https://github.com/mudler/vllm.cpp/issues/907)), `test_backend_cross_device` (`:2063` MoeSiluMul and `:2601` MoE combine/gate vs the CPU oracle, 80205/80207), `test_llama_embedding_fold` (`:254` engine vs direct at `epsilon(1e-5)`), `test_mtp_depth` (`:738` `CHECK(st.capture_shapes == 0)`), `test_qwen3_dflash2_draft` (`:2574` `CHECK(r.generate_threw.empty())`), and `test_ops_attention_dense_fa2` (`:692` `CHECK(Mismatches(on, ref) > 0)` -> `0 > 0`, which reads as a TEST arch-assumption: the case asserts the knob-ON path DIFFERS from the scalar reference, and with `fa2` DISABLED for `[110]` they are the same kernel). Four of the six are CUDA-vs-CPU numeric comparisons off by about one ulp, so they MAY be one sm_110 difference wearing four names -- a hypothesis from the logs, not a measurement. Also folded in: `test_gguf_device_fit_reach`, red and unattributed since 2026-08-15 (`:463`, moved from `:278`, assertion unchanged); and `test_serve_low_tools`, whose `(name, mode)` pair never moved while its CAUSE changed completely -- `73ada0df8` (#1661/#1662) fixed the [#961](https://github.com/mudler/vllm.cpp/issues/961) guard so it skips, the string `shellcheck` appears nowhere in the run's log, and the failure is now four `tests/tools/test_dflash2_speed_harness.py` `ShellDriverTest` cases of 517. The CONTROL was rerun and now falsifies the old conclusion: the baseline reads `FAILED (failures=3, errors=1, skipped=1)` and the control with `shellcheck` 0.9.0 installed reads `FAILED (failures=3, errors=1)` -- the same four cases, differing only in the vanished skip. **That vanished `skipped=1` is a second proof**: the one test that skipped in the baseline is the `shellcheck` guard itself, so with the binary present it stopped skipping and PASSED, showing the instrument was the only thing the install changed. #961 was CLOSED COMPLETED on 2026-08-23 acting on that prompt -- `73ada0df8` fixed its guard while referencing the sibling filing #1661/#1662, which had left #961 orphaned rather than resolved. Not in scope here, because they have owners and did not worsen: the four FA-2 refusals, `test_platform`/`test_op_parity` hardcoding GB10, `test_linear_method` and the three `qwen3_5_gdn_spec_routing` tests (#907), the FP8 pair ([#1725](https://github.com/mudler/vllm.cpp/issues/1725), which improved `SEGFAULT` -> `Failed`), and [#962](https://github.com/mudler/vllm.cpp/issues/962), which reproduced byte-identically at `bitdiff=15/32768`. Table and diff in [`environment.md`](environment.md); the re-measure debt itself is [#955](https://github.com/mudler/vllm.cpp/issues/955) | bug | | [#1791](https://github.com/mudler/vllm.cpp/issues/1791) | `SPEC-DSPARK` | **`scripts/dspark-paired-e2e.sh`'s `settle()` can never break early, so a wait for the GPU to drain always spends its full 360 s however fast the box actually drains.** The same idiom as [#1734](https://github.com/mudler/vllm.cpp/issues/1734), found by sweeping `scripts/` for it: `grep -c .` with an `\|\| echo 0` fallback makes `$n` the two-line string `0\n0`, so `[ "$n" -eq 0 ] && break` answers `integer expression expected` and returns 2 instead of deciding. The BUSY half of the guard works -- a positive count exits 0 and the fallback does not fire -- so only the FREE half is dead, and the failure is in the safe direction, which is why it was paid in silence. This is the FIFTH diagnosis of the idiom in this tree: `scripts/cpu-x86-llamacpp-floor.sh` already carries the removal and the reason in a comment, and a comment in one file is not reachable from another. FIXED IN FLOW with #1734: `\|\| true` keeps grep's own `0` and swallows only its status. The recurrence gate is `TheIdiomIsGoneFromEveryShellScript` in `tests/scripts/test_ltx25_ab_memwatch.py`, which sweeps every `scripts/*.sh` for a counting `grep`/`pgrep` paired with an `\|\| echo` fallback outside a comment; run against `27d8bfa70` it names all three live instances, this one included. It is a TRIPWIRE and says so: it reads text, and a `wc -l` with the same fallback walks past it | bug | +| [#1796](https://github.com/mudler/vllm.cpp/issues/1796) | `SPEC-DFLASH2` | **[#1456](https://github.com/mudler/vllm.cpp/issues/1456)'s conclusion is retracted AT THE ARTIFACT, off-GPU, and the DFlash2 speed ratio's denominator inherits the retraction: `0.8016987337853048` MAY BE FLATTERING US.** #1456 measured a real source build at `CUDA_ARCHS=12.0` emitting `sm_80` for `_vllm_fa2_C` and `sm_75` for `_vllm_fa3_C`, and concluded the GB10 oracle has no `FLASH_ATTN` denominator. **The arch measurement stands; the conclusion does not, and no lease was needed to show it.** Both staged oracle wheels were opened with `zipfile` and walked as fatbinaries on the CPU dev box: `0.1.dev1+g66e5414c6` (this row's oracle, `/mnt/nas_share/rc/oracle-dflash2/`) and `0.1.dev1+g555967922` (the parity pin, `/mnt/nas_share/rc/oracle-vllm/`) each carry 76 FA2 fatbinaries and 192 FA3 fatbinaries, and **every one holds a PTX image beside its SASS image** -- arch 80 for FA2, arch 75 for FA3. The first FA2 PTX payload is zstd and decompresses to `.version 9.0` / `.target sm_80` for `flash_fwd_hdim128_bf16_causal_sm80`. That is the `+PTX` half of `FA2_ARCHS "8.0+PTX"` and it is the mechanism by which the module CAN reach sm_121 -- the artifact establishes a necessary condition, forward-JITtable code being shipped, and that the JIT then ran is an inference from that plus a run selecting `FLASH_ATTN` and generating; `cudaErrorUnsupportedPtxVersion` is the OPPOSITE failure, PTX ISA newer than the driver, which `.version 9.0` under driver 580.173.02 is not. vLLM says it directly too: `FlashAttentionBackend.supports_compute_capability` returns `capability >= DeviceCapability(8, 0)` (`vllm/v1/attention/backends/flash_attn.py:251-252` in the staged wheel). **THE DOWNSTREAM IS THE THING TO CHASE.** #1456's body records that the DFlash2 speed gate's denominator "will be vLLM pinned to `TRITON_ATTN`, by developer decision on 2026-08-20", explicitly because of that conclusion. AGENTS.md requires vLLM's PRODUCTION configuration as the denominator, and on this box vLLM's own auto-selection picks `FLASH_ATTN`: the 0.8017 gate run's log carries BOTH paths in one process, the forced branch taking `TRITON_ATTN` for the 27B target (`cuda.py:426`) and the auto branch taking `FLASH_ATTN` out of four valid backends (`cuda.py:486`), in `/mnt/nas_share/rc/dflash2-1673/out-n1673b/m-gate.log`. **So if `TRITON_ATTN` is the slower backend -- which vLLM's own priority ordering IMPLIES rather than states, and which nothing here measures -- then 16.279 tok/s is too LOW and the ratio is too HIGH. An error in our own favour is the one nobody chases, so it is recorded with its sign.** The ratio is NOT withdrawn and no denominator is substituted, because a wave must not replace a denominator the developer declared. **The five FA layers of [#1685](https://github.com/mudler/vllm.cpp/issues/1685) fall out of the same read**: `vllm/v1/worker/gpu/spec_decode/dflash/utils.py:31-46` sets the draft's `backend=speculative_config.attention_backend` UNCONDITIONALLY, so a harness that sets only the engine backend leaves the draft at `None` and the draft re-runs auto-selection; the siblings `dspark/utils.py:24-28` and `gemma4/speculator.py:66-89` both carry the target's backend through, and dspark's comment names this hazard by name. NOT reconciled in flow, and the reason is precise: what remains is ONE lease running vLLM against itself on this workload with `attention_backend=FLASH_ATTN` against `TRITON_ATTN`, each read back off the built engine, and **no GPU lease is authorised for this session**, so it is PENDING. Recommended collapse, recommended and not taken: #1456, [#1679](https://github.com/mudler/vllm.cpp/issues/1679) and #1685 observe ONE mechanism on two wheels, and #1685 is the one attached to a live number and should survive. Owed under `## Owed` O33 of [the DFlash2 spec](specs/dflash2-spec-decode.md), with caveat 5 beside the ratio in the same file | verification | diff --git a/.agents/specs/dflash2-spec-decode.md b/.agents/specs/dflash2-spec-decode.md index 264ec80fb..4e1de151b 100644 --- a/.agents/specs/dflash2-spec-decode.md +++ b/.agents/specs/dflash2-spec-decode.md @@ -2889,6 +2889,130 @@ list items. difference as a defect is the failure this row keeps having, so it is written down rather than filed. +- **O33 — the denominator is NOT vLLM's production configuration, and the + constraint that justified substituting one is RETRACTED AT THE ARTIFACT.** + Owner: `SPEC-DFLASH2` for this row's ratio, the developer for the declaration. + [#1796](https://github.com/mudler/vllm.cpp/issues/1796) carries this entry and + its evidence, [#1456](https://github.com/mudler/vllm.cpp/issues/1456) is the + retracted premise, and [#1685](https://github.com/mudler/vllm.cpp/issues/1685) + is the observation it explains. + **NOTHING IS RE-MEASURED HERE AND NO DENOMINATOR IS SUBSTITUTED.** + + O22 records that a W6 RUN falsified #1456's conclusion. This entry records that + the WHEEL ITSELF falsifies it, off-GPU, so the retraction no longer rests on + reading a log that was lost with its lease. Both staged oracle wheels were read + with `zipfile` and a fatbinary walk on the CPU dev box — no lease, no GPU, no + CUDA toolkit: + + | wheel | module | fatbins | images per fatbin | arch | + |---|---|---:|---|---:| + | `0.1.dev1+g66e5414c6` (this row's oracle) | `_vllm_fa2_C.abi3.so` | 76 | ELF **and PTX** | 80 | + | `0.1.dev1+g66e5414c6` | `_vllm_fa3_C.abi3.so` | 192 | ELF **and PTX** | 75 | + | `0.1.dev1+g555967922` (the parity pin) | `_vllm_fa2_C.abi3.so` | 76 | ELF **and PTX** | 80 | + | `0.1.dev1+g555967922` | `_vllm_fa3_C.abi3.so` | 192 | ELF **and PTX** | 75 | + + Every fatbinary carries a PTX image beside its SASS image. The first FA2 PTX + payload is zstd, and it decompresses to `.version 9.0` / `.target sm_80` for + `flash_fwd_hdim128_bf16_causal_sm80`. **That is the `+PTX` half of + `FA2_ARCHS "8.0+PTX"`, and it is the mechanism by which the module CAN reach + sm_121.** Be exact about what that buys: the artifact establishes a NECESSARY + condition, that forward-JITtable code is shipped. That the JIT then ran is an + inference from the PTX being there and a run selecting `FLASH_ATTN` and + generating. #1456 read the SASS arch and concluded the module "cannot target + sm_12x"; the arch reading is right, and the conclusion drops the PTX, which is + enough to retract it. + `cudaErrorUnsupportedPtxVersion` is raised when PTX ISA is NEWER than the + driver, and `.version 9.0` under driver 580.173.02 is not that case. vLLM says + the same thing in its own words: `FlashAttentionBackend.supports_compute_capability` + returns `capability >= DeviceCapability(8, 0)` at + `vllm/v1/attention/backends/flash_attn.py:251-252` in the staged wheel, so 12.1 + is a capability upstream declares supported. + + **WHAT THAT DOES TO THE RATIO.** #1456's body records the denominator decision + in one sentence: the DFlash2 speed gate's denominator "will be vLLM pinned to + `TRITON_ATTN`, by developer decision on 2026-08-20 ... it is NOT vLLM's default + backend on this device, and any ratio taken against it must say so." AGENTS.md + requires vLLM's PRODUCTION configuration as the denominator. On this box that + configuration selects `FLASH_ATTN`, and the gate run's own log shows both paths + in the same process: the forced path took `TRITON_ATTN` for the 27B target + (`out-n1673b/m-gate.log:30`, `cuda.py:426`, the branch that honours an explicit + request) and the auto path chose `FLASH_ATTN` out of four valid backends + (`m-gate.log:57`, `cuda.py:486`). So the denominator ran vLLM's sixteen + full-attention target layers on a backend vLLM itself ranks below its first + choice, on a premise that no longer holds. + + **SAY THE DIRECTION PLAINLY: THE ERROR, IF IT IS ONE, IS IN OUR FAVOUR.** If + `TRITON_ATTN` is the slower backend — which vLLM's own priority ordering + IMPLIES rather than states, and which nothing here measures — then the + denominator + 16.27918250335551 tok/s is too LOW and `0.8016987337853048` is too HIGH. An + error that flatters us is the one nobody chases, so it is written down beside + the number rather than left to be noticed. **The ratio is not withdrawn and no + replacement is asserted.** What is asserted is that its denominator rests on a + retracted premise, and that the exposure has a sign. + + **WHAT IS OWED, AND IT IS PENDING A LEASE THIS SESSION DOES NOT HAVE.** One run + of vLLM against itself on this identical workload — same wheel, same host, same + k, same prompts, same `max_num_seqs` — with `attention_backend=FLASH_ATTN` + against `attention_backend=TRITON_ATTN`, each read back off the built engine as + O22 requires. That measurement decides whether 0.8017 stands, is flattered, or + is conservative. It also needs the developer to revisit the 2026-08-20 + declaration, because a wave must not substitute a denominator the developer + set. + + **AND THE FIVE FA LAYERS INSIDE THE DENOMINATOR ARE NOW EXPLAINED, which #1685 + left open as three readings.** + `vllm/v1/worker/gpu/spec_decode/dflash/utils.py:31-46` builds the draft's + config with `backend=speculative_config.attention_backend`, UNCONDITIONALLY: + the target's `attention_config.backend` is not carried through. The harness set + the engine backend and not `speculative_config.attention_backend`, so the + draft's backend was `None`, which is the auto-selection branch of + `CudaPlatform.get_attn_backend_cls` (`vllm/platforms/cuda.py:429-496`), and it + chose `FLASH_ATTN`. Those are the five `model.layers.64-68.self_attn.attn` in + `evidence/vllm-arm.json`. **Two sibling speculators in the same wheel DO carry + the target's backend through** — `dspark/utils.py:24-28`, whose comment names + this exact hazard ("None re-runs backend auto-selection for the draft, which + can pick a different attention class than the target; fall back to the + target's"), and `gemma4/speculator.py:66-89`. So the DFlash path is the + un-defended case rather than an upstream intent, which makes #1685's reading 1 + wrong AS INTENT — the inference is about intent, drawn from two siblings that + defend against exactly this. Reading 2, "those layers fall back at runtime", is + refuted on two counts: `FlashAttentionImpl.__init__` logged `Using + FlashAttention version 2` at `flash_attn.py:906-914`, so an FA implementation + was CONSTRUCTED for those layers, and the runtime half has no fallback to take + — `flash_attn.py` in the staged wheel contains no `fallback` token at all, and + `FlashAttentionImpl.forward` (line 970) raises `NotImplementedError` rather + than degrading. Reading 3 is what the artifact supports. + + **THE RETRACTION INHERITS INTO THREE FILES, AND THIS IS THE ONLY ONE OF THEM + THAT CAN CARRY IT.** `grep -rln FA_USABLE . --exclude-dir=.git` returns + `.agents/benchmark-record.md`, this file, and `.agents/issue-index.md`. The + index quotes `FA_USABLE=0` as a live constraint in the #1456, #1658 and #1685 + rows, and it is append-only by rule and by + `scripts/check-issue-index-append-only.py`. + `.agents/benchmark-record.md:3` self-declares "Append-only forensic record", + and it carries `FA_USABLE=0` un-annotated inside the live "#1685 (new, OPEN)" + item of the 2026-08-22 entry; note that its append-only status is a convention + rather than a gate, which is the whole subject of + [#1373](https://github.com/mudler/vllm.cpp/issues/1373). So this entry is + where a reader lands instead, and the two append-only sites stay as written. + **That `benchmark-record.md` item is STALE rather than wrong** — it says + `FA-CONSTRAINT.txt` RECORDS `FA_USABLE=0`, which is still true, and closes + "Unresolved.", which has stopped being true — so nothing there needs + retracting and the retraction rides the next `SPEC-DFLASH2` entry APPENDED to + that file. Annotating it in place would take a lock on the one file whose own + issue says every appending pull request conflicts, to add a forward pointer. + + A grep of `.agents/oracles/` and `.agents/upstream-sync.md` for `FA_USABLE`, + `FLASH_ATTN` and `TRITON_ATTN` exits 1 with no output, so no oracle file needs + retracting; the plan in #1456's body to write the constraint into + `.agents/oracles/vllm.md` was never carried out. + + **WHAT IS STILL NOT PROVEN IS THE KERNEL LAUNCH.** Construction is proven and + coherent output is measured on both arms. A trace showing an FA2 kernel enter + the SM on this box is not in hand and needs a lease. It is not needed for the + retraction, and it IS needed before anybody claims the forward JIT is free. + ## Now **W6 TOOK THE GATES on 2026-08-21, on `dgx:gpu0` through an `rc` lease, and G2 @@ -3198,7 +3322,7 @@ own leg boundaries since O32, the folded legs are 3.744 s to 10.385 s each, and four ~200-290 s cold legs are discarded by name. The loads sat outside every span. -**FOUR CAVEATS TRAVEL WITH THE RATIO, and they are the reason it is recorded +**FIVE CAVEATS TRAVEL WITH THE RATIO, and they are the reason it is recorded rather than claimed.** 1. **[#1673](https://github.com/mudler/vllm.cpp/issues/1673) FIRED AND DID NOT @@ -3224,6 +3348,16 @@ rather than claimed.** still costs a lease. 4. **One measured axis is not a speed gate.** Memory, TTFT and per-token latency remain open gaps on this row. +5. **[#1456](https://github.com/mudler/vllm.cpp/issues/1456) IS RETRACTED, so + caveat 2 grew a SIGN: the denominator may be flattering us.** `TRITON_ATTN` + was declared for this box because #1456 concluded the wheel's flash-attention + cannot reach sm_12x. Both staged wheels carry `.target sm_80` PTX beside the + SASS, and vLLM's own `supports_compute_capability` admits `>= 8.0`, so the + premise is gone and `FLASH_ATTN` is what vLLM's production configuration + selects here. If it is also the faster one, then 16.279 tok/s is too low and + 0.8017 is too high. Nothing is re-measured, the ratio is not withdrawn, and + [#1796](https://github.com/mudler/vllm.cpp/issues/1796) with `## Owed` O33 + carries what it would take to settle it. **AND O32 IS WHAT MADE THIS RUN EMIT A NUMBER AT ALL.** The whole-window summary the sampler wrote, `evidence/clock-ours.json`, reads 550 busy of 2943 samples —