Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .agents/issue-index.md
Original file line number Diff line number Diff line change
Expand Up @@ -507,6 +507,7 @@ rather than merged. `scripts/check-agent-record.py` gates both.
| [#1474](https://github.com/mudler/vllm.cpp/issues/1474) | `VT-CONV1D-F32-ACC` | `vt::Conv1d` and `vt::ConvTranspose1d` accumulate every output cell in **f64** against a reference that accumulates **f32**, and the provenance recorded for that width is wrong in two independent ways. **vLLM owns no such op** at the parity pin `555967922` — no `Conv1dLayer`, no `ConvTranspose*Layer`, and the only `ConvTranspose` string in the tree names `torch.nn` classes inside an NVTX annotation hook (`vllm/utils/nvtx_pytorch_hooks.py:47-49`); vLLM deliberately drops the vocoder it would otherwise own (`qwen3_omni_moe_thinker.py:1975`, `skip_prefixes=["talker.", "code2wav."]`), and where it DOES own a convolution it states this polarity itself (`csrc/cpu/mamba_kernels.hpp` — `// Accumulate in float32 for precision`). So the reference is torch, and torch was MEASURED rather than read: a 27-tap `[+1e8, 0.1 x 25, -1e8]` probe over a uniform input, where an f32 accumulator lands on exactly `0.0` in ANY order and an f64 one on ~2.5, returns **0.0** for `F.conv1d` at f32 AND at bf16, **0.0** for `F.conv_transpose1d` at f32, and 2.4999998510 at f64 (torch 2.11.0+cu130). **The recorded justification fails twice.** `.agents/specs/minimax-music3.md` §13.2 and §18.3 say f64 "is what every committed golden for all four consumers was taken with", but all three generators run torch in f32 — `gen-bigvgan-goldens.py:48` builds f64 then `.float()`, `gen-ltx2-vae-goldens.py:223,234` and `gen-minimax-music3-acoustic-goldens.py:81,134` cast every parameter and input with `astype(np.float32)` — so the goldens are the output of an **f32-accumulating** reference and the f64 arm has been wider than its own oracle since the op landed. And `include/vt/ops.h` clause (1) cites "every committed golden under `tests/parity/goldens/`", a directory whose 101 entries contain **no** vocoder, BigVGAN, LTX-2.5 VAE, FVQ or general-conv1d golden at all; those live in `tests/vllm/models/*_goldens.inc`. An uncheckable citation is how the first claim survived. **The precedent is landed inside one of the same models**: [#1008](https://github.com/mudler/vllm.cpp/issues/1008) narrowed LTX-2.5's conv VIDEO VAE f64->f32 on this exact argument (`d1b0ea3a8`, [PR #1036](https://github.com/mudler/vllm.cpp/pull/1036)), leaving LTX-2.5 shipping f32 video decode and f64 audio decode in ONE model. The measured risk is the summation ORDER, not the width: in #1008 naive-serial f32 pushed one arm to 5.00679e-06 against a 5e-06 tolerance, repaired by per-input-channel BLOCKED partials — which is what torch's blocked-GEMM convolution does and what `vt::Conv3d` already declares as its contract. Spec [`vt-conv1d-f32-accumulator.md`](specs/vt-conv1d-f32-accumulator.md); corrects both false records in the same flow | bug |
| [#1501](https://github.com/mudler/vllm.cpp/issues/1501) | `SPEC-DFLASH2` | **The row recorded DFlash2's T>0 walk as "inverse CDF" from its opening brief; upstream draws GUMBEL-MAX at BOTH pull-request heads it has cited.** `.agents/specs/dflash2-spec-decode.md` `## Upstream chain` mechanism 2 said "At T>0 the walk is by inverse CDF and returns q over the K candidates", and the `SPEC-DFLASH2` engine-matrix row said the same. At `19c9351904df4c63042671bc67a866ca48dc7d6f` the non-greedy branch of `_selector_walk_kernel` drew `uniform = tl_rand32(gumbel_seed, candidates, includes_zero=False)`, `noise = -tl.log(-tldevice.log1p(-uniform))` and took the argmax of `scores / temperature + noise` — Gumbel-max; at `66e5414c6d75a8529473d977f7458c140bbab8a0` that branch is replaced by one call to `gumbel_noised_argmax`, the same draw. No inverse-CDF walk exists upstream. BLAST RADIUS is a mis-scoped wave rather than a shipped defect: W4 ships the GREEDY arm, which is byte-for-byte upstream's `SAMPLE_PROBABILISTIC=False` arm, and the noised arm is not ported (`ParseSpeculativeConfigJson` refuses `draft_sample_method: "probabilistic"` by name against an accept-iff-equal verify) — but an implementer scoped to write an inverse-CDF walk would have written the wrong algorithm with no oracle to catch it, because the acceptance gate that would notice is itself owed. FIXED IN THE SAME FLOW by W4: both records corrected in place rather than annotated, `## Risks/decisions` D13 records why the noised arm is unreachable here, and `## Owed` O12 records its layout and the Triton Philox stream (`tl.randint4x`, Philox 4x32-10, keyed by the candidate token ids) a bit-parity port would need. Also corrected there: the six-item enumeration of the speculator head move was verified item by item against the two blobs and is correct, but it was missing a SEVENTH change in the base class (`DraftModelSpeculator.__init__` now calls a virtual `draft_logits_spec`) and the `+24 / -6` file it lives in was absent from the delta table. Listed under `## Risks/decisions` D13 in [dflash2-spec-decode.md](specs/dflash2-spec-decode.md) | bug |
| [#1490](https://github.com/mudler/vllm.cpp/issues/1490) | `VT-MATMUL-FP8-BLOCK-CUDA` | **`tests/vt/test_ops_matmul_fp8_block_cuda.cpp` asserted in six places that it has never run against a device, and the loudest is its own header.** The suite ran UNPATCHED on `dgx:gpu0` (NVIDIA GB10, driver 580.173.02, compute capability 12.1) in an `rc` lease on 2026-08-20, at tree `7481a2eecbb26b3d5c977e8707b0384994caf136` — an ancestor of `main` — reporting `5 cases / 136 assertions / 0 failed`, `REFERENCE_TIER_LINES=0` and `TEST_RC=0`, against a device-free baseline of 41; recorded in `5870cb2bf` (PR [#1472](https://github.com/mudler/vllm.cpp/pull/1472)) and confirmed in [#1437](https://github.com/mudler/vllm.cpp/issues/1437). The header opened "THIS FILE HAS NEVER RUN AGAINST A DEVICE" and added that "no number produced here appears in any document as a measurement" — doubly false, because the 5/136/0 result appears in both the public documentation (`docs/USAGE.md` at the time, `docs/models/qwen3-8-27b.md` after [#1491](https://github.com/mudler/vllm.cpp/pull/1491) moved it) and the row's spec. The other five were G2's comment block and the four `NO CUDA DEVICE ... #1189 M5's on-hardware leg is OWED, not passed` `MESSAGE` strings in G2, G7, G8 and G9. The four messages are CORRECT in what they do — on a device-free host the case genuinely did not run, and a skip is not a pass — so the repair is that they say the case did not run HERE and name where the on-hardware result is recorded, not that they are deleted or weakened. Three caveats survive unweakened: NO token gate ([#1189](https://github.com/mudler/vllm.cpp/issues/1189) still owns `Qwen/Qwen3.8-27B-FP8` against the pinned oracle), NO speed claim of any kind, and correctness established on the SEVEN shapes actually run rather than on a class; DSV3's `kv_a_proj_with_mqa` `N=576` capability gap stands as a CUTLASS sm120 limitation. Same class as [#1396](https://github.com/mudler/vllm.cpp/issues/1396) and [#1411](https://github.com/mudler/vllm.cpp/issues/1411), which cover other anchors of the same row; no checker here compares a comment against the measurement it annotates. Listed under `## Owed` in [`vt-matmul-fp8-block-cuda.md`](specs/vt-matmul-fp8-block-cuda.md) | bug |
| [#1520](https://github.com/mudler/vllm.cpp/issues/1520) | `ROAD-V1-QUICKSTART` | **`check-doc-checkpoint`'s `LANDING_SOURCE_FILES` has no entry for the page the README hands its quickstart to, so shrinking `README.md` into a pointer is refused.** The gate refuses a README claim change that touches no landing source, and the set names `.agents/mission.md`, `CMakeLists.txt`, three `benchmarks/demo/*.json` files and the two example mains. Its message names two failures, "routine checkpoints belong in the purpose-specific docs" and "co-edited public projections never justify README churn", and both are a README that GROWS. `row/DOCS-QUICKSTART-1281` is the opposite: it adds `docs/QUICKSTART.md` and cuts the README `## Quickstart` block from three command fences to a four-line pointer, so the README loses material to the purpose-specific document. At head `1b6e458c8` `scripts/agent-preflight.sh` reports 88 gates ok, zero skips and one failure, `doc-checkpoint range`, on a branch that honestly touches no member of the set. FIXED HERE by adding `docs/QUICKSTART.md` to `LANDING_SOURCE_FILES`, on its own branch with its own spec, red-first case and three mutations, because the previous implementer on the quickstart row correctly refused to widen the set inside the change it unblocks. The widening admits ONE exact path and no class: `docs/BUILD.md`, `docs/STATUS.md` and every other page still cannot license a README claim change, and `landing_page` still permits a README edit rather than demanding one. Spec [doc-checkpoint-landing-source-quickstart.md](specs/doc-checkpoint-landing-source-quickstart.md). | bug |
| [#1517](https://github.com/mudler/vllm.cpp/issues/1517) | `ENG-HF-MODEL-DOWNLOAD` | **Every container image this tree produced, and every published CUDA archive, shipped the HuggingFace fetch silently downgraded to plain http.** `docker/Dockerfile` added `libssl3` to the RUNTIME stage for #1280 and never added `libssl-dev` to a BUILDER stage, so `builder-toolchain` and `builder-cuda` installed `binutils build-essential ca-certificates cmake curl file git ninja-build python3` and nothing else. `CMakeLists.txt:2440-2482` answers a failed `find_package(OpenSSL)` with a DOWNGRADE rather than a failure -- `Could NOT find OpenSSL (missing: OPENSSL_CRYPTO_LIBRARY OPENSSL_INCLUDE_DIR)`, then `VLLM_CPP_OPENSSL=ON but OpenSSL was not found; HTTPS support is disabled` and `VLLM_CPP_HF_DOWNLOAD=ON but no transport layer security is available` -- so the build stayed green and `ldd` on the shipped `vllm-server` named only `libstdc++ libm libgcc_s libc`. The runtime `libssl3` was installed and nothing linked it, and the comment at `docker/Dockerfile:175` claiming the server links libssl and libcrypto was false for every image the file produced. NOT a network defect: from inside that same container `ffmpeg -i https://huggingface.co/api/models/does-not-exist/nope/refs` completed a TLS handshake and returned HTTP 401. `.github/workflows/release.yml`'s `cuda_x86` and `cuda_arm64` build inside `nvidia/cuda:13.3.0-devel-ubuntu24.04` with the identical package list, so the PUBLISHED CUDA archives were in the same state; `cpu_x86`, `cpu_arm64` and `vulkan_x86` had transport layer security only because the `ubuntu-latest` runner image happens to preinstall `libssl-dev`, which is an accident nothing defended. Found by `scripts/validate-container-image.py` on its first ever real execution (`hub reach: this image cannot speak HTTPS`), which is the measurement `.agents/specs/hf-model-download.md` recorded as owed from W5. Fixed in flow: `libssl-dev` named in both builder stages and in all five apt-based release lanes, and `scripts/check-build-runtime-deps.py` added to refuse the class -- a stage or lane that compiles the server without the development files, or a runtime that carries `libssl3` with no builder half | bug |
| [#1521](https://github.com/mudler/vllm.cpp/issues/1521) | `ENG-RELEASE-CONTAINERS` | **`docker/reset-stale-build-cache.sh` stamps only the compiler, CMake and CUDA, so a changed DEPENDENCY SET leaves a poisoned `CMakeCache.txt` in the BuildKit cache mount.** Measured 2026-08-20 at `d189f66dd` while verifying #1517. Two `cpu` lane image builds from one tree, differing only in whether the builder stages install `libssl-dev`, share `id=vllm-cpp-container-cpu-$TARGETARCH`. The first configured `-- Found OpenSSL: /usr/lib/x86_64-linux-gnu/libcrypto.so (found version "3.0.13")`. The second, with the package removed, should have taken the documented downgrade at `CMakeLists.txt:2479`; instead the retained cache still held `OPENSSL_CRYPTO_LIBRARY` and `OPENSSL_INCLUDE_DIR`, `find_package` reported found with an EMPTY version (`-- HuggingFace download: HTTPS through OpenSSL (system, dynamic)`), and generate died with `Target "vllm" links to: OpenSSL::SSL but the target was not found`. The stamp is `c++ --version`, `cmake --version` and optionally `nvcc --version` plus the resolved `/usr/local/cuda`, none of which moved, so the reset never fired. LOWER severity than #1517 because the failure is LOUD and CI builds from a cold cache, but it silently invalidated the red arm of a red-to-green measurement and presented a third failure mode that reads like a code defect, which is the broken-instrument shape `.agents/verification.md` names. Worked around by giving the control build its own cache mount id. Found while fixing #1517 and deliberately not fixed in that flow: it moves cache-invalidation semantics and needs its own spec and fresh review | bug |
| [#1511](https://github.com/mudler/vllm.cpp/issues/1511) | `ENG-HF-MODEL-DOWNLOAD` | **The first fetch this tree ever made from `huggingface.co` failed on its first hop, and the refusal blamed a variable the operator had never set.** `--model Qwen/Qwen3-0.6B` printed `HF_ENDPOINT '/api/resolve-cache/models/Qwen/Qwen3-0.6B/<commit>/config.json?...&etag="f5c3703b..."' has no scheme` and downloaded nothing. TWO defects, both properties of what the hub ANSWERS rather than of what it is asked, and the reported one is not sufficient. (1) The `Location` is an ABSOLUTE-PATH REFERENCE, which RFC 7231 section 7.1.2 permits, and both redirect loops in `downloader.cpp` assigned it to `current` unresolved, so the next hop parsed a path as an address. (2) The redirect's own `Content-Length` was read as the size of the file it names: measured on 2026-08-20, the HTTP 307 answering `resolve` for a file that is NOT in large-file storage carries `content-length: 234`, the length of its own text body, and no `x-linked-size`, so a 726 byte `config.json` probed as 234 bytes and the transfer refused on a disagreement with the tree listing. Fixing only (1) reaches the second hop and then refuses the file for a size it never had. WHY FOUR REVIEW ROUNDS DID NOT SEE EITHER: the redirect loops had NO hermetic instrument at all. Every case in `test_downloader.cpp` was served from ONE address, so `IsRedirect` never returned true in a test, and the row's only redirect fixture, `test_hf_hub.cpp`'s `a redirected API answer is refused and the token does not follow it`, serves an ABSOLUTE `Location` and exists to prove the API client REFUSES a redirect rather than follows one. A loop with full line coverage and no case reads as covered. REPAIRED by `HfResolveUrl` (RFC 3986 section 5: absolute, network-path, absolute-path, relative-path and query-only references, query and fragment carried through, dot segments removed) and by reading `Content-Length` only from a non-redirect answer while `X-Linked-Size` keeps its meaning on one. #1485's credential rule is untouched: the manual loop stays, `set_follow_location` stays off, the token is still dropped at the first hop, and the measured large-file-storage redirect crosses to `us.aws.cdn.hf.co` carrying its own signature. Six new cases in `test_downloader.cpp`, each written from a header capture of the live hub. LIVE PROOF on 2026-08-20, x86_64, TLS through system OpenSSL, empty `HF_HOME`: 1.5 GB fetched, `model.safetensors` landing at exactly the 1503300328 bytes the hub declared and `config.json` at 726 rather than 234, the server reaching `listening on http://0.0.0.0:<port>` and answering `/v1/completions` with `" Rome. The capital of France is Paris. The capital of Germany is Berlin."` Spec [`hf-model-download.md`](specs/hf-model-download.md) | bug |
Loading
Loading