diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md index 4bb1091de..8a786f138 100644 --- a/.agents/benchmark-record.md +++ b/.agents/benchmark-record.md @@ -24838,3 +24838,221 @@ The job builds `ltx2-gen` from `7b9e207b1` in-lease, refusing to proceed unless `bin/` and `job/` are staged under `/workspace/ltx25-fullmodel` on the house NAS. Run evidence is `out/20260819T150230Z/` with `PROVENANCE`, `memguard.tsv`, `phases.tsv`, `verify.txt` and `render.log` per rung. +--- + +## MUSIC3-VOCODER-CONV — the vocoder decode window, dependent-chain vs tiled f64 accumulators, Jetson Thor sm_110 (2026-08-19, `row/MUSIC3-VOCODER-CONV-SPEED`, #672, #1334) + +### What was measured, and what it is not + +`music3::VocoderDecode` at the shipped `MiniMaxMusic3VocoderConfig` geometry — +the call `Music3DecodeChunks` brackets as the `vocoder.decode_window` profile +bucket (`minimax_music3_speech.cpp:397-398`) — over a sweep of latent window +lengths, on the DEFAULT arm (`VLLM_CPP_VOCODER_DEVICE` unset, so the host +kernel). Driver `tools/bench/music3_vocoder_conv_ab.cpp`, recipe +`scripts/music3-vocoder-conv-ab.sh`. + +It is NOT an end-to-end synthesis. The weights are synthetic, so nothing here is +a claim about audio, and **no checkpoint was read** — the staging assertion +`SRC_BYTES == DST_BYTES` is therefore NOT APPLICABLE rather than skipped, because +there is no path to assert. The e2e pair on the real checkpoint is owed +(`minimax-music3.md` §18.9). + +It takes **no GPU clock window** and it is a CPU measurement, so nothing here is +quotable as a per-kernel or cross-box figure. + +**53.6 s is the CUDA arm.** §15.2's profile ran `VLLM_CPP_VOCODER_DEVICE=cuda`, +so the 53.6 s / 33.2 %-of-wall figure that defined this row is the DEVICE arm's +decode window, while the default is `cpu` (§13.6) and that is the arm this row +moves. No projection from 1.42x onto 53.6 s is made, because that would multiply +two different arms. + +### The lease, and why it is the whole of the serialisation + +`rc` job `da3a2f94-90e3-4e97-b519-9456310673b7` on **`thor:gpu0`**, +`--max-runtime 150m`. Worker `rc-worker-hqfj4`, `Linux 6.8.12-1021-tegra` +aarch64, 14 cores. No `ssh`, no `rc hold`, no `$GPU_LOCK`. §13.10's whole speed +axis is VOID because its arms went in over `ssh` under the file mutex while +another session held the same box through `rc`, so the two mutexes did not +exclude each other; this run has exactly one. + +### The arms — two trees, two build directories, two binaries + +Both built INSIDE the lease from two clones under `/tmp` (local overlay, not the +CIFS `/workspace`), `CMAKE_BUILD_TYPE=Release` (`-O3`), CPU-only, `ninja -j 8`. + +| | before | after | +|---|---|---| +| ref | `f06b9e93d` (kernel only) | `c16bacc13` = `row/MUSIC3-VOCODER-CONV-SPEED` | +| `src/vt/cpu/cpu_conv1d_general.cpp` sha256 | `6fb15174c1533b93b1bc58401e3a21fe0f4da9101729108d89118ee72fb146db` | `a0e429ace35364798dfa060f08002941544b3136a59c62712778df1c2174dba1` | +| `vllm_music3_vocoder_conv_ab` sha256 | `d90e3912cd636666067df089b0267fbd428300cac80a29706113693b46610b22` | `41ba78d2b7a8b99ef34a32fd6620b1530c88ac1cc279a811d338482543b811b5` | + +`diff -rq` over `src/` names that ONE file as the only difference, and the recipe +HARD-FAILS (`exit 5`) when the two binaries hash the same. That guard is there +because the depth row's first Thor pair was VOID for precisely this — both arms +were one binary, and the tell was identical call counts, since equal times are +noise where equal binaries are identity (§16.6a). + +**The `after` ref `c16bacc13` was later rewritten** (the branch was rebuilt to +fold `docs/USAGE.md` into the commit that owed it, because +`check-doc-checkpoint` validates each commit individually). The kernel is +BYTE-IDENTICAL across that rewrite — `a0e429ace3536479...` at both +`c16bacc13` and the branch head — so the measurement is unaffected, and this is +recorded rather than left for a reader to discover that the sha is unreachable. + +### Correctness, established before any speed number was read + +On the after arm, in the same container. `test cases`, `assertions` AND `Status` +are quoted for each, because `assertions: 0` is a skip wearing a pass. + +| suite | result | rc | +|---|---|---| +| `test_ops_conv1d_general` | 9 cases, 375 assertions, 0 failed, `SUCCESS!` | 0 | +| `test_host_parallel` | 8 cases, 877 assertions, 0 failed, `SUCCESS!` | 0 | +| `test_vocoder1d` | 10 cases, 58 assertions, 0 failed, `SUCCESS!` | 0 | +| `test_bigvgan` | 6 cases, 65 assertions, 0 failed, `SUCCESS!` | 0 | + +Three `[SKIP]` lines printed and are read rather than ignored: this worker has +**no `nvcc`**, so the build is CPU-only and every CPU-vs-CUDA arm in that file +did NOT run. The device provider is untouched by the change and §18.3's argument +says its `memcmp` must still hold, but an argument is not a measurement, and the +re-measurement is OWED. + +(That probe also CORRECTS §13.10's record of this worker: it carries `gcc`, +`g++`, `cmake`, `ninja`, `make`, `python3` and `git`, and only `nvcc` is +missing. §13.10 recorded "no compiler and no toolchain at all" and named a +worker image as the blocker for re-measuring under a lease. For a CPU arm that +blocker does not exist.) + +### The sweep — arms ALTERNATED, 3 rounds, best-of-3 per point + +`uptime` load average **3.29 before the build, 9.07 before the sweep, 8.48 +after** it, 0 other users. The box was NOT idle; both arms ate the same +contention, which is what alternating them is for. + +Medians of the three rounds, in seconds: + +| latent frames | before | after | ratio | s/latent after | +|---|---|---|---|---| +| 20 | 5.5688 | 4.0831 | **1.364x** | 0.2042 | +| 40 | 11.0535 | 7.8186 | **1.414x** | 0.1955 | +| 86 | 23.5149 | 16.6614 | **1.411x** | 0.1937 | +| 172 | 47.9201 | 33.6498 | **1.424x** | 0.1954 | +| 344 | 97.4463 | 67.7083 | **1.439x** | 0.1974 | + +The loudest pair is KEPT and not dropped: 20 frames is the weakest ratio in the +set, and it is the size §15.9 priced the device arm at. Round-to-round spread +within an arm is under 1 % at every point (e.g. before@344: 97.1298 / 97.4463 / +97.9561; after@344: 67.4177 / 67.7083 / 67.9122), so the 1.36-1.44x band is well +clear of the noise. + +**Flat across a 17x span of work**, which is the signature of a RATE change +rather than of a fixed overhead — the same test §13.10 applied to the device arm, +run in the other direction. + +### The arms are BIT-IDENTICAL at full scale + +The driver prints an FNV-1a fingerprint over the whole stereo waveform. Across +all six arm-rounds every length produced ONE value on both arms: + +| latent frames | fingerprint (both arms, all rounds) | +|---|---| +| 20 | `0x7c31c2ea73418503` | +| 40 | `0x35a02aad4c9cb983` | +| 86 | `0xc2d5eaf095d1c483` | +| 172 | `0x2dc69976150a5903` | +| 344 | `0x95e771d5f0051283` | + +Six processes, two binaries, one answer per length. A sum would let two different +waveforms agree; a byte-wise hash cannot. + +### The x86 half, labelled as what it is + +Design-selection measurements, single-threaded, `taskset -c 2`, AMD Ryzen 9 +9950X3D, g++ 13.3.0, `-ffp-contract=off`, min of 5, `memcmp`-identical on every +case. These SELECTED the design; they do not gate it, and they carry no clock +attribution. + +| stage (per stereo stream, latent 20) | MMAC | shipped | tiled | | +|---|---|---|---|---| +| `conv_in` k7 1024->1536 | 220.2 | 0.1096 | 0.0322 | 3.40x | +| `b0.res.conv1` k7 768 | 660.6 | 0.3289 | 0.0612 | 5.37x | +| `b1.res.conv1` k7 384 d3 | 1321.2 | 0.6630 | 0.1260 | 5.26x | +| `b1.res.conv2` k1 384 | 188.7 | 0.1585 | 0.0245 | 6.48x | +| `b2.res.conv1` k7 192 | 1321.2 | 0.6760 | 0.1238 | 5.46x | +| `b3.res.conv1` k7 96 | 660.6 | 0.3281 | 0.0613 | 5.35x | + +Every ratio in the table above is a KERNEL figure at one stage geometry, not a +decode-window figure. The window also runs `vt::ConvTranspose1d`, the alias-free +activations, the strided downsamples and the threadpool around all of them; the +window ratios on this row are 1.36-1.44x (14 threads, §18.8a) and 2.16x (one +thread, §18.8b), and the review that asked for this distinction measured the real +project build at `-O2` single-threaded and got 2.56x / 2.67x on the window +against the 5.2-5.8x the kernel gives there. The two quantities are never +multiplied or substituted. + +The shipped kernel's rate is **1.76-1.86 GMAC/s on one core** of the 20-core Zen +5 — 1.86 / 1.82 / 1.84 at ranks 8 / 16 / 32 and 1.7622 at rank 1024 for the depth +decoder's loop of the same shape, recorded in the MUSIC3-DEPTH entry above, and +1.827 for THIS loop at `in_per_group = 384`, `kernel = 7` under GCC 13.3 `-O2 +-ffp-contract=off`, pinned, measured by this row's fresh review on 2026-08-19. + +**The cycles-per-MAC conversion this entry first carried is WITHDRAWN.** It read +"1.7-2.0 GMAC/s ... ~2.8-3.0 cycles per multiply-accumulate on a 5.0 GHz Zen 5 +whose `fadd` latency is 3", which is not self-consistent — 2.0 GMAC/s at 5.0 GHz +is 2.5 cycles per MAC, and a strictly dependent chain of latency-3 `fadd`s cannot +exceed 5.0 / 3 = 1.67 GMAC/s at all, so the top of that band sat above its own +ceiling. The deeper problem is the instrument: the x86 host is a KVM guest +reporting a nominal 4291.948 MHz with no `cpufreq` interface and +`perf_event_paranoid` at 4, so neither its boost clock nor a cycle count is +observable from inside it, and every cycles-per-MAC number here was a conversion +through an assumed clock. What survives is the arithmetic identity — a dependent +chain of latency-`L` adds cannot beat `clock / L` MACs per second — and the fact +that breaking the chain, changing nothing else about the arithmetic, its order or +its width, is worth 2.16x per core. `minimax-music3.md` §18.2 carries the full +correction. + +`vt::ConvTranspose1d` measured separately: at `-O3` the compiler already +vectorises its scatter and the op is **~6 % of the chain's wall** against +`vt::Conv1d`'s ~94 %; at `-O2` it does not, and the fixed tap chunk is worth +2.7-2.9x on the three kernels of 8 taps or more. + +### The single-thread pair: the KERNEL is 2.16x, the THREADPOOL returns 1.37x + +Second lease, `rc` job `5b98f95e-a37b-4fa2-8ee9-81959caa828f` on `thor:gpu0`, +`--max-runtime 45m`, a FRESH container with both arms rebuilt from the same two +refs into a NEW pair of binaries (`a3b14f2995...` before, `5b894d6b67...` after), +so this is an independent build as well as an independent run. Latent length 20, +the point where the 14-thread ratio is weakest: + +| threads | before | after | ratio | +|---|---|---|---| +| **1** (`VLLM_CPP_CPU_THREADS=1`) | 37.0508 s | 17.1751 s | **2.157x** | +| 14 (default), same container, same binaries | 5.4845 s | 4.0182 s | **1.365x** | + +The 14-thread control reproduces the main run's 1.364x to three digits on +different binaries, which is what makes the single-thread leg comparable rather +than merely adjacent. Fingerprint `0x7c31c2ea73418503` on every leg of both runs. + +Scaling, which is where the cause is: + +| arm | 1 -> 14 threads | of a possible 14x | +|---|---|---| +| before | 6.76x | 48 % | +| after | **4.27x** | **31 %** | + +Neither arm scales and the FASTER one scales worse — the signature of a shared +resource that the tiled kernel reaches sooner because it needs the same bytes in +less time. This does NOT identify the resource: no bandwidth counter was read and +none is available on this worker, so "memory bandwidth" is the leading candidate +and not a finding. What it does establish is that the next lever is the +vocoder's PARALLEL DECOMPOSITION and not the kernel: `ForOutputRows` partitions +output channels, so all 14 threads sweep the whole input tensor. + +It also settles the aarch64-vs-x86 question posed by the headline ratios: per +core the same source is worth 2.16x here against ~5x on AVX-512, a gap the 4x +narrower f64 vector explains without anything further. + +### No parity claim + +SGLang-Omni is still `gateable = no`. Every reference axis in `docs/BENCHMARKS.md` +stays `PENDING`. Everything above is an internal two-arm number on one named box. diff --git a/.agents/issue-index.md b/.agents/issue-index.md index 7bbf5378e..d25b7da21 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -431,6 +431,7 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#1325](https://github.com/mudler/vllm.cpp/issues/1325) | `ENG-RECORD-ANCHOR-RATCHET` | `scripts/record-anchor-baseline.json` stores a `"total"` that no code reads. `load_record_anchor_baseline` in `scripts/check-agent-record.py` returns `{bucket: int(data["buckets"][bucket]) for bucket in RECORD_ANCHOR_BUCKETS}` with `RECORD_ANCHOR_BUCKETS = ("stale", "broken")`, and it is the file's only reader; `check_record_anchors` iterates those two buckets, and `write_record_anchor_baseline`'s refusal compares `result.total > sum(previous.values())` — the buckets, not the stored `total`. Measured at `af87251c5`: mutating `"total": 38` to `39` (a file whose total disagrees with `32 + 6`) leaves `check-agent-record.py` exit 0, tree restored byte-for-byte by sha256. So this row's own budget file carries the exact shape the row exists to name: a recorded figure no gate reads, sitting beside the figures that are read and presenting as if it were checked. `--write-baseline` compounds it by printing `-> 38`, which reads as the value it stored and is the one no later run consults. TWO candidate resolutions, deliberately not chosen here because choosing belongs to the fixing row: read it and assert `total == stale + broken` on load, or drop the field and derive it at read time, which is the shape AGENTS.md §Records prefers. Either is a semantic change to `check-agent-record.py` owing a spec and a red-before case in `tests/scripts/test_agent_record.py` `RecordAnchorRatchet`. Distinct from [#1287](https://github.com/mudler/vllm.cpp/issues/1287) and [#1270](https://github.com/mudler/vllm.cpp/issues/1270), neither of which reaches the unread field. Also under `## Owed` in [`record-anchor-ratchet.md`](specs/record-anchor-ratchet.md) | bug | | [#1316](https://github.com/mudler/vllm.cpp/issues/1316) | — | `scripts/main-baseline.py` renders a scheduled run that executed ZERO jobs as `RED` with all 11 covered jobs `missing`, so `NEWEST BASELINE: RED at ` names a tree the run never checked out. Measured at `origin/main` `250db75a2`: runs `32206456661` and `32140419182` both return `startedAt: null` for every job, because GitHub cancelled them while they were pending in the single `ci-schedule-refs/heads/main-mudler/vllm.cpp` group, whose queue holds one run ([#274](https://github.com/mudler/vllm.cpp/issues/274)). Fail-closed, and the `missing (expected, never ran)` line is accurate about the jobs; the defect is the verdict word, because a run that executed nothing is NOT RUN rather than RED, and the newest verdict should fall through to the newest run that actually ran. NOT fixed in flow: `test_an_expected_job_the_payload_never_mentions_is_red` and `test_a_narrowed_run_reports_red_and_names_what_never_ran` deliberately assert missing-is-red so a narrowed run cannot pass, and separating "narrowed" from "never started" changes what the verdict means, which owes its own spec, red-before evidence and a fresh reviewer. Owed under `## Owed` of [baseline-lane-eviction.md](specs/baseline-lane-eviction.md), which removes the only observed producer of a zero-job run | bug | | [#1314](https://github.com/mudler/vllm.cpp/issues/1314) | `SPEC-DFLASH2` | **DFlash2 (`DFlash2DraftModel`) is unported**, and one config rule would run the published checkpoint wrong in silence. Upstream carries DFlash2 as a SECOND architecture beside DFlash rather than as a change to it ([vllm#52816](https://github.com/vllm-project/vllm/pull/52816), OPEN at head `19c9351904df4c63042671bc67a866ca48dc7d6f`, base `9842d701`, 755+/5-, 11 files, plus the stacked guard fix [vllm#52883](https://github.com/vllm-project/vllm/pull/52883)): DFlash1 gains two subclass seams and keeps every behaviour, and the new architecture adds a GROUPED DYNAMIC DEPTHWISE CONVOLUTION around each attention and each MLP sublayer plus a CANDIDATE SELECTOR that replaces the independent per-slot argmax with a scored path walk over the target head's top-K. Shapes taken from the published checkpoint rather than from the diff: `z-lab/Qwen3.8-27B-DFlash2`, safetensors header range-read 2026-08-19, 81 tensors -- DFlash1's set plus `layers.N.{attention,mlp}_conv.{base_kernel (2,2,5120), kernel_projection.weight (1280,5120)}` x5 and `candidate_selector.{hidden_projection.weight (256,5120), predecessor_codebook, successor_codebook}` at `(248320,256)` bf16 each, ~254 MB resident the DFlash1 lane never allocates; `conv_kernel_size 2`, `conv_group_size 16`, `selector_rank 256`, `selector_top_k 16`, `block_size 8`. **The silent one:** that config declares all five layers `sliding_attention` AND `is_causal false`, while our resolution mirrors the OLD upstream rule (causal iff SWA, unless `dflash_config.causal`, `include/vllm/model_executor/models/qwen3_dflash.h:22-24`), so every layer would run CAUSAL -- plausible tokens, a token gate against our own output sees nothing, and only ACCEPTANCE moves, which the lossless verify hides. Upstream changes `_dflash_layer_causal` to read `is_causal` first, in the same commit. Three further things are owed and none of them is silent: no route for the `DFlash2DraftModel` architecture string (the same classification code as the open `DSparkDraftModel` gap, [#1193](https://github.com/mudler/vllm.cpp/issues/1193)); no top-k that EMITS the surviving (id, value) pairs, where the decision is to extend the sort-free pivot-bracket threshold search already ported from the same FlashInfer approach at `src/vt/cuda/cuda_sample.cu:297-506` rather than port FlashInfer's 3380-line general radix kernel; and the path walk must run ON DEVICE from the first landing, because the identical sequential shape in DSpark shipped host-side and measured 28% of the 27B draft step ([#436](https://github.com/mudler/vllm.cpp/issues/436)) before `SampleSequentialDevice` moved it. Already reusable unchanged: `vt::DFlashBlockAttention`, the DFlash runner/rejection/GDN-rollback lane, and the loader's target-shared `embed_tokens`+`lm_head`, which is already what a DFlash2 checkpoint needs. BEYOND-PIN by developer decision 2026-08-19 (mirror the open PR now, reconcile if review moves it), in the same posture `SPEC-DSPARK-QWEN3-ROUTING` takes toward vllm#52197; the parity pin `555967922` does not carry the architecture and is NOT advanced. Gate arm is bf16 27B plus the GGUF drafter arm in the same wave, oracle = vLLM built at the PR head, acceptance measured SAME-TRAJECTORY because `SPEC-DFLASH` D8 spent a campaign on a divergent-trajectory confound that D9 refuted. Spec [dflash2-spec-decode.md](specs/dflash2-spec-decode.md) | feature | +| [#1334](https://github.com/mudler/vllm.cpp/issues/1334) | `MODEL-MUSIC-minimax-music3-mini-max-music3-for-conditional-generation` | MiniMax-Music3's vocoder is **53.6 s of a 161.6 s run (33.2 %)** after [#1238](https://github.com/mudler/vllm.cpp/issues/1238), against 12.0 % before it, and #1238's depth A/B measured it at 53.6 s on BOTH legs — it is a FIXED term that grows as a share of every other improvement. **Both arms are slow for one cause.** `vt::cpu::Conv1dKernel` (`src/vt/cpu/cpu_conv1d_general.cpp::Conv1dKernel`) computes each output cell with ONE f64 accumulator swept over `(ic ascending, k ascending)`, which for a Music3 residual unit is `384 * 7 = 2688` STRICTLY DEPENDENT f64 additions per output element: no instruction-level parallelism, no vectorisation at any width, and a measured 1.7-2.0 GMAC/s per core = ~2.8-3.0 cycles per multiply-accumulate on a 5.0 GHz Zen 5 whose `fadd` latency is 3. The CUDA provider loses by a DIFFERENT mechanism with the same cause (§15.9: 3.552 s device vs 2.983 s host at latent length 20; §13.10's per-stage ratios flat at 0.37-0.40x across a 150x span of work): one accumulator per cell means it is not latency-bound but f64-RATE bound, and Thor's consumer Blackwell runs fp64 at a fraction of its fp32 rate. **The f64 stays.** §13.2 records it as what all four consumers' goldens were taken with and what makes the CUDA provider `memcmp`-identical; narrowing to torch's f32 re-gates four shipped models and is left OWED. **The chain breaks bit-identically instead**: one f64 accumulator per cell over a TILE of output positions with the `(ic, k)` sweep hoisted outside, so every cell receives the identical sequence of IEEE-754 double additions of the identical products in the identical order, interleaved across independent cells rather than serialised — a scheduling change, not an arithmetic one, so `memcmp` survives by construction. Measured single-threaded at the real vocoder shapes, `-ffp-contract=off`, `memcmp`-identical on every case: **3.4x-6.5x** per stage. Spec [`minimax-music3.md`](specs/minimax-music3.md) §18 | perf | | [#1339](https://github.com/mudler/vllm.cpp/issues/1339) | `ENG-HF-MODEL-DOWNLOAD` | SUPERSEDES the integrity sentence in the [#1280](https://github.com/mudler/vllm.cpp/issues/1280) row above, which still reads "refuses a listing where two distinct files carry the same identifier". That any-duplicate rule was refuted in review and never shipped: `lfs.oid` is the sha256 of the contents and the plain `oid` is the git blob sha1, so two byte-identical files share one identifier by construction and the rule rejected legitimate repositories. Two narrower rules shipped instead, neither depending on the token. An identifier whose characters are all the same is refused, which is the 17 August 2026 `Lightricks/LTX-2.5` shape. A shared identifier whose entries disagree on a size the listing REPORTED is refused, whether or not their paths differ, and an agreed size is accepted. The second review then found two holes in that size rule and both are repaired here. It kept the first entry carrying an identifier as that identifier's owner whatever its size state and compared only against a known-size owner, so ONE entry reporting no size disarmed the rule for that identifier for the rest of the listing, and `HF_ENDPOINT` is user configurable. The owner is now the first entry whose size was reported, which is enough rather than a list because size equality is transitive. It also carried a distinct-path guard that was pinned by nothing and that accepted one path listed twice at 4096 and 2048 bytes, the exact self-contradiction the rule exists to catch; the guard is removed. `HfFile::size` becomes `std::optional` in the same repair, because `0` spelled both a zero-byte file and an unreported size and W3 sizes a byte range from that field. Spec [`hf-model-download.md`](specs/hf-model-download.md) | bug | | [#1335](https://github.com/mudler/vllm.cpp/issues/1335) | `ENG-CUDAGRAPH-BREAK` | W5, the LAST migration stage of the break-point capture seam: the three SINGLE-SHAPE drivers move onto `vt::GraphCaptureScope` + `vt::BreakableGraph` in `kFull` — the DFlash draft graph (`qwen3_dflash.cpp`), the DeepSeek V4 decode graph (`deepseek_v4.cpp`) and the Laguna decode graph (`laguna.cpp`), whose own note at `:2116-2119` asked for this seam by name and named V4's driver as the sibling that moves with it. **NINE OF NINE DRIVERS ARE NOW ON THE SEAM**, and a call-shaped grep over `src/vllm/` for `BeginCapture`, `EndCaptureGraph`, `ReplayGraph` and `DestroyGraph`, with comment lines excluded, returns NOTHING — every surviving textual hit is prose: the last three hand-rolled `BeginCapture`/`EndCaptureGraph` pairs, their raw `void*` handles, two hand-written `DestroyGraph` destructors and DFlash's private `try`/drain are gone. The three per-model rollback switches STAY (`VT_V4_DECODE_GRAPH`, `VT_DFLASH_GRAPH`, `VT_LAGUNA_DECODE_GRAPH`) because each is a same-binary A/B lever for exactly one driver; what changes is that `VLLM_CPP_CUDAGRAPH` reaches all three for the first time, through `vt::GraphCaptureEnabled()` inside the scope. **D10, the auxiliary-stream fork/join, is DISCHARGED here and could not have been discharged earlier**: `GraphCaptureScope` now owns the set of side queues forked since the current segment opened and joins every outstanding one before `Backend::EndCaptureGraph`, which is the port of `_end_current_segment` (`breakable_cuda_graph.py:353-361` @ SGLang pin `f63458b5be`) plus the `wait_stream` hook (`:101-153`) whose only purpose is to populate that set. W1's break point was on a model that forks nothing and W2/W3/W4 all opened `kFull`, which has ONE segment and so no segment CLOSE inside a fork window; W5 owns the only driver whose fork is inside the captured region by construction (`laguna.cpp:2572-2576` fork, `:2612` join), and that driver is the rule's production caller through `vt::GraphNoteFork`/`vt::GraphNoteJoin` — no monkey-patch, because our fork is an explicit `RecordEvent`/`QueueWaitEvent` pair rather than an implicit torch call. **The rule is a COUNTER and an ORDER, not prose**, because W3 measured a mode guard that stayed green at 226/226 under the mutation it named: `vt::GraphBreakStats` gains `forks_tracked` and `forks_auto_joined`, and the assertion is the single backend trace `Begin RecordEvent QueueWaitEvent EndCaptureGraph …`, since closing a capture with an unjoined fork FAILS at `cudaStreamEndCapture` and "the join happened BEFORE the close" is the whole claim. Five arms including the CONTROL where the model joins first and the scope must do nothing; two mutations, both compiled clean — deleting `JoinOutstandingForks()` reds only the new case on 5 assertions, and making it over-fire reds it on 8. **DFlash is the one single-shape driver gateable without a GPU, and the reason is its admission predicate**: it names neither a device type nor a kernel registry, while V4's `CanRunResidentDecode` refuses `kCPU` outright and again without the four CUDA-registered V4 kernel families, and Laguna's whole capture class is behind `#ifdef VT_MARLIN_NVFP4` (a CUDA build on a marlin-nvfp4 arch — `sm_12xa` or `sm_110`). So `tests/vllm/models/test_qwen3_dflash_decode_graph_seam.cpp` lands RED-first (3 cases, 0 passed, 16 assertions, 7 failed, exit 1 against the unmigrated driver) and GREEN after (3 cases, 18 assertions), with the G2 mutation restoring the pre-W5 raw pair reddening ONLY that file while `test_breakable_graph` (265), `test_qwen3_decode_graph_seam` (231), `test_qwen3_moe_decode_graph_seam` (228), `test_voxtral_decode_graph_seam` (230), `test_deepseek_v2_decode_graph_seam` (230), `test_qwen3_5_decode_graph_seam` (129) and the driver's OWN `test_dflash_propose` (31) all stay green — that last one being the point, since a driver's bit-exactness suite cannot see which capture machinery ran. **G1 and G2 for V4 and Laguna are OWED on hardware and named rather than implied**, as is G5's ROCm/Tenstorrent arm, which the fleet carries no device for. Still NO throughput claim anywhere: prefill is 3.8% host-idle at >96% GPU-busy, the 27B prefill gap is 92.5% non-GEMM glue, decode is already captured, and the sibling row `ENG-CUDAGRAPH-DEDUP` closed its benefit question negative after measurement | feature | | [#1352](https://github.com/mudler/vllm.cpp/issues/1352) | `ENG-CUDAGRAPH-BREAK` | The DFlash draft step ran TWO full forwards per propose under `VLLM_CPP_CUDAGRAPH=0`. CAUSED by W5's migration ([#1335](https://github.com/mudler/vllm.cpp/issues/1335)) and FIXED IN THE SAME FLOW. `ForwardBlockLogitsWithDeviceKV`'s paged predicate was `UseDflashGraph() && d.b.SupportsGraphCapture() && support_static_graph_mode()`, which was complete while the driver's capture was its own `BeginCapture` pair — nothing outside the file could turn that off. W5 routes the capture through `vt::GraphCaptureScope`, which reads `VLLM_CPP_CUDAGRAPH` itself, so the switch gained a state the predicate did not know about: with it set to `0` on a capture-capable backend the driver still chose the CAPTURE lane, ran its eager warm pass, opened an INERT scope and ran the whole `ForwardPagedBody` a SECOND time inside it — two full draft forwards per propose, forever, because `g_state` could never reach 2. NOT a correctness defect (an inert scope executes the forward eagerly, so the logits are real and identical) and confined to a non-default diagnostic lane, which is precisely why nothing would have found it later. Fix: `vt::GraphCaptureEnabled()` becomes the third conjunct, so the switch selects the driver's existing single-forward eager path, which is what it means for the other eight drivers. **The gate is unusual because the defect is invisible to every ordinary one**: both versions emit IDENTICAL logits and both leave `segments_captured` at 0, so neither a token gate nor the seam's counters can separate them. The assertion is therefore LANE IDENTITY against a live control — a backend that CANNOT capture takes the eager path by construction, so a capture-capable backend with the switch OFF must do exactly the same work — and it runs in a CHILD PROCESS because `vt::GraphCaptureEnabled()` is read once per process into a function-local static. **The first instrument was WRONG and the harness now records why**: counting `Backend::Alloc` read `control=11, switch-off=0`, and would have read those same two numbers whichever lane ran second, because the `DevicePool` serves the second of two identical forwards out of its free list without calling `Alloc` at all — an instrument reporting the ORDER of the runs rather than their shape, which would have failed the CORRECT program. `CaptureCapableCpuBackend` counts `Copy` instead, per-operation work the pool cannot absorb, and gains a `supports_capture` flag so `StaticGraphCpu(false)` produces the control lane. Red-first on the unfixed driver: `DIFFERENT-LANE`, `control copies=4` against `switch-off copies=5`, exit 1, one assertion, compiled clean at 22 insertions and 3 deletions — and that same run reported the logits `IDENTICAL`, which is the measurement showing why nothing else could have caught it. Green after at 4 cases / 23 assertions, with all seven sibling seam gates unchanged | bug | diff --git a/.agents/specs/minimax-music3.md b/.agents/specs/minimax-music3.md index dedc8f5cb..d5fd97c69 100644 --- a/.agents/specs/minimax-music3.md +++ b/.agents/specs/minimax-music3.md @@ -1103,6 +1103,25 @@ under `--speech-device 1`, which §13 routes through `vt::MatmulBT`, `vt::Add` with device-resident weights. Every profile number quoted above is the CPU arm's and still describes it exactly. +**Where the vocoder stands after §18, so this section and `docs/STATUS.md` agree +rather than one of them being newer.** The row does NOT move lifecycle state — +MiniMax-Music3 stays `ACTIVE` — so by AGENTS.md's trigger table neither this line +nor the STATUS row was owed: both are tied to a lifecycle move, and only +`docs/BENCHMARKS.md` is owed by a new accepted measurement. They are written +anyway, and written TOGETHER, because the STATUS row's notes cell carries this +model's measured wins and would otherwise name the depth decoder's and not the +vocoder's. That mismatched pair was a review finding on this row (2026-08-19), +and the resolution recorded here is the pair, not the trigger. + +`vocoder.decode_window` was the term §16.7 named as the largest one no row owned. +It is owned now, and measured: **1.36-1.44x** on Thor's 14 threads and **2.16x** +per core (§18.8a, §18.8b), bit-identical output on every length. Next on this +model, in the order §18.9 argues for: the vocoder's PARALLEL DECOMPOSITION, which +§18.8b measured as the cap rather than assuming one; the CUDA-vs-CPU `memcmp` +arm, which needs a box carrying `nvcc`; the e2e pair on the real checkpoint; and +the f64-throughput explanation of the device arm, which §18.2 now carries as an +inference rather than a finding. + --- ## 10. The parity sweep, the music-only server, and the weights record (#672) @@ -3682,6 +3701,466 @@ gets mutated like any other. --- +## 18. The vocoder's convolution stops running one dependent add chain per cell (#672, #1334) + +§16.7 named this row before it existed: *"The vocoder is now the largest single +term in a run, and no row owns it. At 53.6 s it is 32.9 % of wall after this +change, against 12.0 % before, and it did not move because nothing here touches +it."* This is that row. The issue is +[#1334](https://github.com/mudler/vllm.cpp/issues/1334). + +### 18.1 The gap, and the two facts that shape it + +`vocoder.decode_window` is **53.6 s**, and #1238's depth A/B measured it at +53.6 s on BOTH legs. It is therefore a FIXED term: it does not scale with the +depth work, it was 12.0 % of wall before that row and 33.2 % after, and it grows +as a share of every future improvement to anything else. + +And the device arm is not the answer waiting to be switched on. §15.9 prices it +at **3.552 s CUDA against 2.983 s host** at latent length 20, and §13.10's +per-stage ratios are flat at 0.37-0.40x across a 150x span of work. The arm is +real, reachable, and slower. + +### 18.2 Why both arms are slow, and why it is ONE cause + +`vt::cpu::Conv1dKernel` (`src/vt/cpu/cpu_conv1d_general.cpp::Conv1dKernel`) +computes each output cell with a SINGLE f64 accumulator swept over +`(ic ascending, k ascending)`: + +```c++ +for (int64_t t = 0; t < length; ++t) { + double acc = bp != nullptr ? bp[oc] : 0.0; + for (int64_t ic = 0; ic < in_per_group; ++ic) + for (int64_t k = 0; k < kernel; ++k) { ...; acc += ...; } + on[t] = static_cast(acc); +} +``` + +For a MiniMax-Music3 residual unit that is `in_per_group * kernel = 384 * 7 = +2688` **strictly dependent** f64 additions per output element. The loop has no +instruction-level parallelism and cannot be vectorised at any width, because +every add waits on the previous one. Its measured rate on one core of the 20-core +Zen 5 is **1.76-1.86 GMAC/s**: 1.86 / 1.82 / 1.84 at ranks 8 / 16 / 32 and 1.7622 +at rank 1024 for the depth decoder's loop of the same shape +(`.agents/benchmark-record.md`, the MUSIC3-DEPTH entry), and **1.827 GMAC/s** for +THIS loop at `in_per_group = 384`, `kernel = 7` under GCC 13.3 `-O2 +-ffp-contract=off`, pinned, measured by the fresh review of this row on +2026-08-19. + +**A correction, because the sentence this section first carried was not +self-consistent** (review finding, 2026-08-19). It read "1.7-2.0 GMAC/s on one +core, ~2.8-3.0 cycles per multiply-accumulate on a 5.0 GHz Zen 5 whose `fadd` +latency is 3", and three of those numbers cannot all be true at once: 2.0 GMAC/s +at 5.0 GHz is **2.5** cycles per MAC rather than 2.8-3.0, and a strictly +dependent chain of latency-3 `fadd`s cannot exceed **5.0 / 3 = 1.67 GMAC/s** at +all, so the top of the quoted band sat above its own stated ceiling. + +**The cycles-per-MAC figure is WITHDRAWN rather than corrected, because this box +cannot supply the clock it converts through.** The measurement host is a KVM +guest on a Ryzen 9 9950X3D; `/proc/cpuinfo` reports a nominal 4291.948 MHz, +`/sys/devices/system/cpu/cpu0/cpufreq/` does not exist in the guest, and +`perf_event_paranoid` is 4, so the boost clock during a timed loop is not +observable from inside it and neither is a cycle count. Every cycles-per-MAC +number this section quoted was therefore a conversion through an ASSUMED clock, +which is where the inconsistency came from. What the conversion would give across +the plausible range, stated so the next reader does not redo it: 1.827 GMAC/s is +2.35 cycles per MAC at the reported 4.29 GHz, 2.74 at 5.0, and 3.12 at the "5.7 +GHz-class" this record uses elsewhere for the same box. + +**What the ceiling implies, which is the part worth keeping.** A dependent chain +of latency-`L` adds cannot beat `clock / L` MACs per second, so the measured +1.76-1.86 GMAC/s bounds `L` at or below ~2.4 cycles at the clock the guest +reports, and clears a latency-3 ceiling at any clock below 5.3 GHz. Either the +add latency on this core is under 3 or the core was boosting above 5.3 GHz, and +this box can distinguish neither. **The mechanism does not rest on that +constant.** What supports it is that breaking the chain — and changing nothing +else about the arithmetic, its order, or its width — is worth **2.16x per core** +(§18.8b). A dependency is the only property that intervention removed. + +The CUDA provider loses for a DIFFERENT mechanism, and **the mechanism is an +INFERENCE rather than a finding** — stated that way here because the first +version of this paragraph asserted it flat, and a fresh review was right to +refuse it (2026-08-19). + +What is CHECKED, by reading the source: `src/vt/cuda/cuda_conv1d_general.cu`'s +`Conv1dKernelCudaImpl` gives each output cell its own thread and one f64 +accumulator, and pins every operation with `__dadd_rn` / `__dmul_rn`. So the +device arm is not latency-bound the way the host loop is — there are millions of +independent chains — and its arithmetic is f64 throughout. + +What is MEASURED: §13.10's per-stage device-vs-host ratios are flat at 0.37-0.40x +across a 150x span of work. That rules out a FIXED overhead, which would punish +the smallest stage far more than the largest, and it rules out very little else. + +What is INFERRED, and owed: that the residual is f64 THROUGHPUT — that Thor's +consumer Blackwell runs fp64 at a small fraction of its fp32 rate and the kernel +is sitting on that limit. Nothing here measured it. No fp64:fp32 ratio was taken +on the box, no occupancy or pipe-utilisation counter was read, and no CUDA A/B +was run against an f32-accumulate arm. It is the explanation this row offers, not +a result it establishes, and §18.9 carries it as owed. + +The two arms' mechanisms are different either way. The property they both turn on +— and the only one this row changes on the host — is that the arithmetic is f64 +and arranged as one accumulator per output cell. + +### 18.3 The f64 stays, and it does not have to move + +§13.2 records why the accumulator is f64: it is what the `vocoder1d` host loops +used, therefore what every committed golden for all FOUR consumers was taken +with — MiniMax-Music3, MiniMax-H3's audio VAE, LTX-2.5's audio VAE and +IndexTTS-2.5 — and it is what makes the CUDA provider `memcmp`-identical to the +host. torch accumulates an f32 conv in f32; this is a deliberate, recorded +divergence. + +Narrowing it to f32 is a real lever and it is NOT this row (§18.9). It would +re-gate four shipped models, it cannot inherit §13.4's `memcmp`, and it is worth +strictly less after this row than before it. + +**Because the chain can be broken without touching the width.** Hold one f64 +accumulator per output cell over a TILE of output positions, and hoist the +`(ic, k)` sweep OUTSIDE the position loop: + +```c++ +for (t0 = 0; t0 < length; t0 += TILE) { + for (i = 0; i < tn; ++i) acc[i] = seed; + for (ic ascending) + for (k ascending) + for (i in the in-range part of the tile) acc[i] += x[...] * w[ic][k]; + for (i = 0; i < tn; ++i) on[t0 + i] = (float)acc[i]; +} +``` + +Fix any single output cell `t = t0 + i` and read the additions it receives, in +order: the bias, then `(ic=0,k=0)`, `(ic=0,k=1)`, ... — the identical sequence of +IEEE-754 double additions of the identical double products, in the identical +order, as the shipped loop. Nothing is reassociated. The additions are +INTERLEAVED across independent cells rather than serialised into one, which is +a scheduling change and not an arithmetic one. **`memcmp` equality survives by +construction, and no tolerance is introduced or widened.** + +The same argument is why the zero-padding skip has to be handled by CLAMPING the +tile's position range rather than by testing each position: for a fixed `k` the +in-range `i` form one contiguous interval, so the skipped `(t, ic, k)` triples +are exactly the ones the shipped loop skips. + +### 18.4 The compiler has to be given a constant trip count, and that is measured + +The restructure alone is not the win. GCC's `-O2` vector cost model is +`very-cheap`, which vectorises only a loop whose trip count is a known multiple +of the vector width, and the accumulate loop's bounds are runtime values. So the +kernel takes a fast path with a CONSTANT trip count when the whole tile is +in range and the stride is 1, and a fixed-width chunked path plus a scalar tail +otherwise. + +`Release` is `-O3` and is what CI, `scripts/build-cpu-release.sh` and the +accelerator release build use; `scripts/dgx-bringup.sh` uses `RelWithDebInfo`, +which is `-O2`. The kernel is therefore written to be fast at BOTH, rather than +inheriting whichever one the next measurement happens to use. Measured +difference on the same source: with a runtime-bounded accumulate loop the +speedup is 1.1-1.8x at `-O2` and 5.1-5.4x at `-O3`; with the constant-trip fast +path it is 5.2-5.8x at `-O2` and 5.3-6.5x at `-O3`. + +**Every one of those six figures is the KERNEL alone**, taken on `Conv1dKernel` +at the vocoder's stage geometries, and none of them is what the decode window +does. The window is not only this convolution: it also runs +`vt::ConvTranspose1d` (§18.5 prices it far lower), the alias-free activations, +the strided downsamples that keep the shipped gather, and the threadpool and +allocation around all of them. The gap between the two quantities is measured +rather than asserted — the same review that asked for this clause built the +project at `-O2` and timed the window single-threaded, and got **2.56x and +2.67x** where the kernel gives 5.2-5.8x. §18.8a's **1.36-1.44x** is the same +quantity again at `Release` on 14 threads, and §18.8b's **2.16x** is that window +on ONE thread. A kernel number applied to the window overstates the win by +roughly a factor of two, so the two are never quoted interchangeably here. + +`stride > 1` keeps the shipped arithmetic path. The MiniMax-Music3 vocoder's +`Conv1d` calls are all stride 1; the strided caller is the alias-free downsample +in `vocoder1d::AliasFreeActivation1d::Apply`, which is depthwise +(`in_per_group == 1`) and whose chain is one tap deep, so it is not the shape +this row is about. That is a decision with a reason, not an omission. + +### 18.5 `vt::ConvTranspose1d` needs much less, and the number says why + +The scatter already writes `kernel` INDEPENDENT cells per input value, so it has +ILP by construction and the only thing missing is a constant trip count on the +tap loop. At `-O3` the compiler finds it and the op is **~6 % of the chain's +wall**; at `-O2` it does not, and a fixed-width tap chunk recovers 2.7-2.9x on +the three kernels of 8 or more taps. + +Both are done, because the four-line chunk is what makes a `RelWithDebInfo` +measurement of this row mean the same thing as a `Release` one. Neither changes +the visit order: the taps of one input value land in `kernel` DISTINCT cells, so +chunking them reorders nothing. + +### 18.6 Upstream anchors + +vLLM does not implement a DAC Flow-VAE vocoder, and no secondary oracle is +consulted for a NUMERIC question here, because this row changes no number. The +semantics being preserved are `torch.nn.functional.conv1d` / +`conv_transpose1d` as §13's port already mirrors them +(`minimax_music3_vocoder.py:42,44,55,89,98`), and the arithmetic contract being +preserved is this repository's own, stated in `include/vt/ops.h` at +`vt::Conv1d` / `vt::ConvTranspose1d` and gated by +`tests/vt/test_ops_conv1d_general.cpp` and +`tests/vllm/models/test_host_parallel.cpp`. **The oracle for this row is the +shipped kernel itself, carried verbatim into the gate**, which is the same +instrument §12.2 and §13.3 used and for the same reason. + +### 18.7 Tests and gates + +| gate | what it holds | +|---|---| +| `test_ops_conv1d_general` | the op contract, including the CPU-vs-CUDA `memcmp` arm on a CUDA build | +| `test_host_parallel` | the shipped `vocoder1d` entry points against a VERBATIM copy of the pre-op host loop, bitwise, at five thread counts | +| `test_vocoder1d`, `test_bigvgan` | the two smallest consumers | +| `test_minimax_h3`, `test_ltx2_vae`, `test_minimax_music3_*` | the four consumers' committed goldens | + +The shapes the first two do NOT reach today are what this row's new cases add, +because the restructure's whole risk surface is the tile boundary and the +position clamp: + +- `padding != 0` — `cpu_conv1d_general.cpp:74` states in its own comment that + every existing caller passes `padding == 0` and that the skip "is therefore + unreachable on those shapes and exists for torch parity". A clamp that is + wrong at the left or right edge is invisible without it. +- a `length` that is not a multiple of the tile, and a `length` BELOW one tile. +- `dilation > 1` combined with a tile boundary. +- the catastrophic-cancellation case §13.5 built, at a length that spans several + tiles, so an f64 accumulator stored through an f32 cannot hide a reordering — + **once per SWEEP AXIS, because one case covers one axis and not the other.** + +Every one of them is asserted against a verbatim in-test copy of the pre-change +kernel, bitwise. + +**The two axes, and why the count is two rather than one** (review finding, +2026-08-19). The sweep this row hoists out of its accumulators is +`(ic ascending, k ascending)`, and reversing EITHER loop is a genuine +reassociation. The cancellation case as first written had teeth along `ic` ONLY, +by construction: it pairs input CHANNELS +(`w[(oc*cin+1)*kernel+k] = w[(oc*cin+0)*kernel+k]` with `x[0] = +2^40` and +`x[1] = -2^40`), and its own teeth self-check reverses `ic` and nothing else. +Measured on this tree: with the kernel's `k` loop reversed and nothing else +changed, `test_ops_conv1d_general` reported 9/375, `test_host_parallel` 8/877, +`test_vocoder1d` 10/58 and `test_bigvgan` 6/65, all `SUCCESS!` and all rc 0 — +binary sha256 `c4bb0e76...` against the baseline `760061c5...`, so the mutation +was in the binary that ran and the green was not a stale artifact. + +The hole is OLDER than this row — the pre-change kernel with its `k` sweep +reversed is green too — so it is not a regression this row introduced. It is +still this row's to close, because this row is what turns that order into a +load-bearing guarantee. `tests/vt/test_ops_conv1d_general.cpp` therefore carries +a SECOND cancellation case, "holds its TAP order", which pairs +2^40 against +-2^40 across `k` inside one input channel held constant along its length, and +whose teeth check reverses `k` alone. With that case present the same mutation +fails 1552 of 1576 cells in exactly that case and nowhere else (10 cases / 379 +assertions, 1 failed, binary `eb25d992...`), and the restored tree hashes back to +`f84d3a87...`. `SerialConv1d` grew a `reverse_k` parameter beside `reverse_ic` +for it. + +**So state the guarantee with its axes attached.** What is gated is that the +order each individual cell sees is unchanged along `ic` AND along `k`. Neither +"nothing held the forward sweep's order" nor "a length at which an f64 +accumulator stored through an f32 cannot hide a reordering" was true of both +axes when it was first written here; both were true of `ic` and false of `k`. + +### 18.8 Speed evidence — what it must be, before it is taken + +`.agents/benchmarking.md` and the recorded defects of §13.10 and §16.6a bind +this row to four things it cannot report without: + +1. **An `rc` lease on `thor:gpu0`.** §13.10 is VOID because its arms were taken + over `ssh` under `$HOME/gpu.lock` while another session held the same box + through `rc`. Not a caveat — step zero. +2. **Two separately built binaries, in separate source trees and separate build + directories, whose `sha256` DIFFER.** §16.6a is void because both arms were + the same binary, and the tell was identical call counts rather than equal + times. +3. **The checkpoint staged to local disk**, with `SRC_BYTES == DST_BYTES` + asserted and `findmnt` printed for the path actually read. §15.6 measured + 780 s cold CIFS / 428 s warm against 7.5 s local. +4. **A RANGE of latent lengths.** One point is what made the device arm look + simply "slower" in §15.9. + +### 18.8a The measurement, TAKEN — Jetson Thor, under a lease, two binaries + +**`rc` job `da3a2f94-90e3-4e97-b519-9456310673b7` on `thor:gpu0`**, worker +`rc-worker-hqfj4`, `Linux 6.8.12-1021-tegra` aarch64, 14 cores, +`--max-runtime 150m`, 2026-08-19. No `ssh`, no `rc hold`, and no file mutex — the +lease is the whole of the serialisation, which is the thing §13.10 did not have. + +Both arms built INSIDE the lease from two clones in `/tmp` (local overlay, not +the CIFS `/workspace`), `CMAKE_BUILD_TYPE=Release` (`-O3`), CPU-only, `ninja -j 8`: + +| | before | after | +|---|---|---| +| `src/vt/cpu/cpu_conv1d_general.cpp` sha256 | `6fb15174c1533b93` | `a0e429ace3536479` | +| `vllm_music3_vocoder_conv_ab` sha256 | `d90e3912cd636666` | `41ba78d2b7a8b99e` | + +`diff -rq` over `src/` reports that ONE file as the only difference between the +trees, and the harness refuses to time anything when the two binaries hash the +same. That guard exists because §16.6a's first Thor pair was VOID for exactly +this: both arms were the same binary, and the tell was identical call counts, +because equal times are noise where equal binaries are identity. + +**Correctness first, on the after arm, before any speed number was read** +(`test cases` / `assertions` / `Status` quoted in full, because `assertions: 0` +is a skip wearing a pass): + +| suite | result | rc | +|---|---|---| +| `test_ops_conv1d_general` | 9 cases, 375 assertions, 0 failed, `SUCCESS!` | 0 | +| `test_host_parallel` | 8 cases, 877 assertions, 0 failed, `SUCCESS!` | 0 | +| `test_vocoder1d` | 10 cases, 58 assertions, 0 failed, `SUCCESS!` | 0 | +| `test_bigvgan` | 6 cases, 65 assertions, 0 failed, `SUCCESS!` | 0 | + +Three `[SKIP]` lines are printed and are read rather than ignored: this worker +has no CUDA toolkit (`nvcc` MISSING), so the build is CPU-only and every +CPU-vs-CUDA arm in that file, including this row's new one, did NOT run here. +The device provider is unchanged by this row, and §13.4's `memcmp` arm was +measured on a CUDA build; **it has not been re-measured against the tiled host +kernel, and that is named as owed in §18.9 rather than implied by these greens.** + +**The sweep: arms ALTERNATED, three rounds, best-of-3 per point, the default +arm** (`VLLM_CPP_VOCODER_DEVICE` unset, so this is the host kernel — see below +for why that matters). `uptime` load average 3.29 before the build, 9.07 before +the sweep, 8.48 after it; 0 other users; the box was NOT idle and both arms ate +the same contention, which is what alternating them is for. + +Medians of the three rounds, in seconds: + +| latent frames | before | after | ratio | +|---|---|---|---| +| 20 | 5.5688 | 4.0831 | **1.364x** | +| 40 | 11.0535 | 7.8186 | **1.414x** | +| 86 | 23.5149 | 16.6614 | **1.411x** | +| 172 | 47.9201 | 33.6498 | **1.424x** | +| 344 | 97.4463 | 67.7083 | **1.439x** | + +The loudest pair is KEPT rather than dropped: 20 frames is the weakest ratio in +the set and it is the one §15.9 priced the device arm at. The spread across a +**17x span of work is 1.364x to 1.439x**, which is the signature of a RATE change +and not of a fixed overhead — the same test §13.10 applied to the device arm, run +in the other direction. + +**And the arms are BIT-IDENTICAL at full scale.** The harness prints an FNV-1a +fingerprint of the whole stereo waveform, and across all six arm-rounds every +length produced one value on both arms: `0x7c31c2ea73418503` (20), +`0x35a02aad4c9cb983` (40), `0xc2d5eaf095d1c483` (86), `0x2dc69976150a5903` (172), +`0x95e771d5f0051283` (344). Six processes, two binaries, one answer per length. + +**What this number is NOT.** It is the decode window's COMPUTATION at the shipped +geometry with synthetic weights, driven through `VocoderDecode` — the call +`Music3DecodeChunks` brackets as `vocoder.decode_window` +(`minimax_music3_speech.cpp:397-398`). It is not an end-to-end synthesis, no +checkpoint was read, and therefore §18.8's staging assertion +(`SRC_BYTES == DST_BYTES`, `findmnt` on the path read) is NOT APPLICABLE here +rather than skipped: there is no path to assert. The e2e pair on the real +checkpoint is owed (§18.9). + +**And 53.6 s is the CUDA arm, not this one.** §15.2's profile ran +`VLLM_CPP_VOCODER_DEVICE=cuda`, so the 53.6 s / 33.2 % that defined this row is +the DEVICE arm's decode window. The default is `cpu` (§13.6) and that is the arm +this row moves. Applying 1.42x to the 53.6 s figure would be comparing two +different arms, so no such projection is made here. + +### 18.8b The KERNEL is 2.16x and the THREADPOOL gives back a third of it + +The obvious reading of §18.8a is that aarch64 simply vectorises worse than +x86 and 1.42x is what the kernel is worth here. **That is measured, and it is +wrong.** + +Second lease, **`rc` job `5b98f95e-a37b-4fa2-8ee9-81959caa828f` on `thor:gpu0`, +`--max-runtime 45m`**, a fresh container, both arms rebuilt from the same two +refs into two new binaries (`a3b14f2995...` before, `5b894d6b67...` after — both +different from the first run's pair, so this is an independent build as well as +an independent run). One latent length, 20 frames, because that is where §18.8a's +ratio is weakest and therefore where the claim is hardest: + +| threads | before | after | ratio | +|---|---|---|---| +| **1** (`VLLM_CPP_CPU_THREADS=1`) | 37.0508 s | 17.1751 s | **2.157x** | +| 14 (default), same container, same binaries | 5.4845 s | 4.0182 s | **1.365x** | + +The 14-thread control reproduces §18.8a's 1.364x to three digits, on a different +container and a different pair of binaries, which is what makes the single-thread +figure comparable to it rather than merely adjacent. + +**Read the scaling instead of the ratio and the cause is plain:** + +| arm | 1 -> 14 threads | of a possible 14x | +|---|---|---| +| before | 6.76x | 48 % | +| after | **4.27x** | **31 %** | + +Neither arm scales, and the FASTER one scales WORSE. That is the signature of a +shared resource: the tiled kernel needs the same bytes in less time, so it +saturates whatever is shared sooner and gives back a third of its per-core win. +The kernel is worth **2.16x on this box**; the threadpool returns 1.37x of it. + +Stated with its limits. This does not identify the resource — no bandwidth +counter was read, and none is available on this worker — so "memory bandwidth" +remains the leading candidate and not a finding. What it does establish is where +the next lever is, and it is not the kernel: **it is the vocoder's parallel +decomposition**, which partitions OUTPUT CHANNELS and therefore has every thread +sweep the whole input tensor. §18.9 carries it as owed rather than as a ceiling. + +And it removes one hypothesis §18.8a left open: the aarch64 codegen is NOT the +whole story, because per core the same source is worth 2.16x here against ~5x on +AVX-512 x86 — a gap the 4x narrower f64 vector explains without needing anything +else. + +### 18.9 What is OWED after this row, named rather than left to a profile + +- **The f32-accumulate variant** (§13.10 step 3) is untouched and is now worth + less, because the f64 path costs several times less than it did. It still + needs its own gate against each of the four consumers' goldens and cannot + inherit §13.4's `memcmp`. +- **The device arm's staging** (§13.6's owed list: device-resident weights, one + persistent queue, a chain that stays on the device between stages) is + untouched. This row does not make the device arm win, and after it the host + bar the device arm has to clear is several times higher. +- **`stride > 1`** keeps the shipped arithmetic path (§18.4). +- **`ltx2_audio_vae.cpp:75`'s own 2-D host convolution loop** still routes + through no op at all (#1114), so it is not reached by anything here. +- **The CUDA arm's f64-THROUGHPUT explanation is unmeasured** (§18.2). The flat + 0.37-0.40x ratio rules out a fixed overhead; it does not establish what the + residual IS. Three instruments would, none of them run here: an fp64:fp32 rate + probe on `thor:gpu0`, a pipe-utilisation or occupancy counter over + `Conv1dKernelCudaImpl`, and a CUDA A/B against an f32-accumulate arm — which + the f32 variant in the first bullet would supply for free. Until one of them + exists the sentence in §18.2 is an inference, and this row does not act on it. + Deliberately NOT settled in flow: it needs a `thor:gpu0` lease and this row's + measurement budget was spent on the host arm it actually moves. +- **The CUDA-vs-CPU `memcmp` arm has NOT been re-measured against the tiled + kernel.** The Thor worker carries `gcc`, `g++`, `cmake`, `ninja` and `python3` + but **no `nvcc`** (probed 2026-08-19, which also CORRECTS §13.10's "no compiler + and no toolchain at all"), so the arms above are CPU-only and every + CPU-vs-CUDA case printed `[SKIP]`. The argument that it must still hold is + §18.3's — the host arm's per-cell order is unchanged, and §13.4's device kernel + was written against that order — but an argument is not a measurement. It needs + a build on a box with a CUDA toolkit. +- **The e2e pair on the real checkpoint.** §18.8a is the decode window's + computation with synthetic weights, so it prices the stage without pricing a + song. The e2e leg is what would carry the staging assertion and a WAV identity + check, in the shape §16.6b took for the depth row. +- **The vocoder's PARALLEL DECOMPOSITION is now the lever, and §18.8b measured + it rather than guessing.** The kernel is worth **2.16x per core** on Thor and + the threadpool returns **1.37x**, because both arms scale badly (6.76x and + 4.27x of a possible 14x) and the faster arm scales worse. `ForOutputRows` + partitions OUTPUT CHANNELS, so every one of the 14 threads sweeps the whole + input tensor for its own slice of channels; a decomposition that also splits + the TIME axis would give the threads disjoint input windows. Not attempted + here, and not a ceiling: what it needs first is a bandwidth counter, and this + worker has none. + +### 18.10 Stop conditions + +Stop and report rather than widen scope if: the byte-exactness gates cannot be +made green without a tolerance; `thor:gpu0` is unhealthy or the lease cannot be +taken; or the measured host win does not survive the threadpool, which would +mean the parallel arm is bound by something this row did not measure. +--- + ## 19. The depth decoder reaches the device (#672, [#1309](https://github.com/mudler/vllm.cpp/issues/1309)) — §11.4's LAST owed device row, and the dtype it was blocked on §11.4 named three device rows. The vocoder closed in §13, the DiT in §14, and diff --git a/CMakeLists.txt b/CMakeLists.txt index 45ac66734..1327a010e 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -2362,6 +2362,42 @@ foreach(_music3_ab_arm before after) endforeach() target_compile_definitions(vllm_music3_depth_stage_ab_after PRIVATE MUSIC3_AFTER) +# ── The MiniMax-Music3 vocoder decode-window A/B driver (#672, #1334) ───────── +# `tools/bench/music3_vocoder_conv_ab.cpp` is the artifact behind spec §18's +# numbers, and it is registered as no test, so nothing in `ctest` runs it. +# +# WHY THIS ONE IS AN EXECUTABLE WHERE ITS SIBLING 20 LINES ABOVE IS AN OBJECT +# LIBRARY. That divergence was a review finding (2026-08-19) and it is argued +# here rather than left to be noticed twice. The depth driver's two arms are +# `#if defined(MUSIC3_AFTER)` branches of ONE file, so compiling it is the whole +# rot guard and linking `main` would only add an entry point nobody calls; an +# OBJECT library is strictly cheaper and it says so. +# +# This driver's arms are two SOURCE TREES, the shape spec §18.8 requires and that +# §16.6a was voided for not having, and the recorded recipe is a SHIPPED SCRIPT: +# `scripts/music3-vocoder-conv-ab.sh` configures two clones, runs +# `ninja ... vllm_music3_vocoder_conv_ab` in each, `sha256sum`s the two results +# and REFUSES to time anything when they hash the same, then executes them with +# `--lengths=` / `--repeats=`. Every one of those steps needs a linked binary. +# An OBJECT library would guard the compile and break the reproduction, which +# inverts the point: the rot this target exists to prevent is a §18 nobody can +# re-run. +# +# What it costs, stated rather than hoped: one ~21 MB executable in a build tree, +# no test registration, and no `install(TARGETS ...)` entry — see the install +# list below, which names `vllm` and `vllm_shared` only — so it does not ship and +# no packaged artifact changes. `main` is linked and never runs in CI; the +# allocation the depth driver's comment worries about happens on invocation, not +# at load. +add_executable(vllm_music3_vocoder_conv_ab tools/bench/music3_vocoder_conv_ab.cpp) +target_link_libraries(vllm_music3_vocoder_conv_ab PRIVATE vllm) +target_include_directories(vllm_music3_vocoder_conv_ab PRIVATE + ${CMAKE_SOURCE_DIR}/include ${CMAKE_SOURCE_DIR}/src ${CMAKE_BINARY_DIR}/include) +target_include_directories(vllm_music3_vocoder_conv_ab SYSTEM PRIVATE + ${CMAKE_SOURCE_DIR}/third_party) +target_compile_features(vllm_music3_vocoder_conv_ab PRIVATE cxx_std_20) +vllm_cpp_set_warnings(vllm_music3_vocoder_conv_ab) + # ── Packaged shared/static libvllm (M3.5 Task 3, library-first packaging) ───── # The static `vllm` core lib above stays as-is (tests + examples link it and # reach internal C++ symbols). Here we ALSO package a SHARED libvllm.so that diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index 63db7296c..a23078d0a 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -524,6 +524,7 @@ built on it rather than keeping the flattering one. | MiniMax-Music3 device arm, Jetson Thor sm_110 (#672) | `--device 1` vs `--device 0`, same request/seed, idle box: 2 AR frames **846.6 vs 835.1 s (1.014x SLOWER)**; 10 frames **1430.4 vs 1512.1 s (0.946x)**. Fit: **-11.65 s/frame, +34.8 s fixed** | A third duration (the fit has no residual), and moving the depth decoder + DiT + vocoder, which are 5 of 6 stages and still host scalar loops | | MiniMax-Music3 DiT device arm, `thor:gpu0` sm_110 (#672) | Per DiT forward at the capture's geometry, same binary/weights/inputs, idle box: **204.955 s host vs 0.186-0.187 s device, 1094-1102x** (1201x fitted). Staged ONCE (0.61 s; loop intercept 0.063 s). Whole process 3.5-4.5x | e2e song pair NOT runnable (host DiT alone ~37.6 h at 30 steps). Depth decoder/condition mix (bf16-storage), vocoder (no `ConvTranspose1d`) still host. Detail: benchmark-record | | MiniMax-Music3 depth stage, x86-64 20-core (#672) | Thor sm_110, real checkpoint, 3 pairs (`7b22b5b0`): `depth_forward` **348.27 -> 78.32 s, 4.45x** (1414 -> 808 calls); wall **446.33 -> 163.00 s, 2.74x**. E2E audio byte-identical, 6 runs. x86 synthetic: 3.50x | Wall is 2.74x not 4.45x: `vocoder.decode_window` is a fixed 53.6 s, now 32.9% of the run and owned by no row. `lm_decode_step` fell 2.65 s unexplained; Thor vs x86 is a HYPOTHESIS. Detail: benchmark-record | +| MiniMax-Music3 vocoder DECODE WINDOW (#672, [#1334](https://github.com/mudler/vllm.cpp/issues/1334)) | Thor sm_110 `rc` lease, host arm, two trees, two binary sha256, 3 alternated rounds: **1.364x at 20 latents to 1.439x at 344**; second lease, 1 thread **2.157x** | Never the conv kernel alone (5.2-5.8x at `-O2`). Fingerprints identical. Not e2e, no checkpoint, so staging is N/A not skipped. 53.6 s is the CUDA arm. CUDA `memcmp` NOT re-measured. Detail: benchmark-record | | MiniMax-Music3 CPU host kernels, x86-64 20-core (#672) | KERNEL A/B at the vocoder's real geometry, min of 5 interleaved rounds: convolution chain **13.36 -> 1.25 s, 10.7x**; `Conv1d` 12.03x, `LinearNoBias` 10.88x. Output fingerprints IDENTICAL on both arms | e2e pair VOID (cold CIFS cache; a foreign `ctest` at load 76.6) and re-running. Stages 0/1 only ~2x: the pivot trades WEIGHT locality for accumulator locality. Detail: benchmark-record | | MiniMax-H3 render coherence (`row/H3-RENDER-CLOSE` #77) | **CLOSED: a COHERENT scene on GB10.** #70/#74 white was wrong-PARTITION usage (t2va on the ref2va ckpt); t2va on the FL2VA GGUF renders a prompt-matched orange cat (adj-cos 0.95 vs 0.06, no patch-grid) | Verified first: t2va inputs byte-exact vs upstream; CUDA device==host at seq 1920. Follow-up `H3-TASK-PARTITION-GUARD`: the task/partition mismatch now RAISES 1:1 with `_resolve_task` (spec §8.6-8.7) | | MiniMax-H3 image conditioning (`row/H3-CONDITIONED-E2E`, `row/H3-VISION-SCATTER`, `row/H3-REF2VA-ASSEMBLY`) | **fl2va COHERENT; ref2va assembly bug FIXED+gated.** vision→cond scatter gated; ref2va block-dim double-division fixed + RED-first gated (128 vs 512) + a permanent ref2va DiT-forward rung (§8.10) | grid RE-ATTRIBUTED: with the fix ref2va grids in fp4 AND bf16, and t2va with no refs on the ref2va NVFP4 also grids while FL2VA-GGUF renders, so it is the **NVFP4 checkpoint/loader**, NOT assembly/fp4 (§8.10) | diff --git a/docs/STATUS.md b/docs/STATUS.md index 04a4df406..46c8ab3f2 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -153,7 +153,7 @@ token-for-token correctness against the pinned oracle. | InternLM2 dense (fused-`wqkv` interleaved split) | Correctness-complete, speed-pending | Token-exact 16/16 (internlm2-chat-1_8b): 12/16 strict + 4/16 bf16 near-tie (max gap 0.0 nats), 0 divergent; first InternLM model; ZERO new compute kernel (reuses the Llama dense forward; the only delta is a loader-side de-interleave of the fused `wqkv`, which packs q/k/v interleaved by KV-group) | | MiniMax-H3 (`MiniMaxH3DiTModel`, video+audio DIFFUSION) | **ABI v12 ONE SURFACE; device selector uses generic `DeviceType`; DSR 32.** t2va+fl2va COHERENT; bf16 shards STREAM | ref2va ckpt fidelity §8.12; encoder A/B §8.15; GB10 re-verify residual; CPU fold 6/137 (one queue + device provenance mutation-gated) | | LTX-2.5 (`LTX2VideoTransformer3DModel`, video+audio DIFFUSION) | **L1-L9c landed (#435).** 21.00B / 48 blocks. `VideoEngine` seam + ABI **v18**, DiT forward (CPU f32 parity, bf16 device-resident), Gemma-4 TE, both VAEs, connector, pipeline, NVFP4/FP8, keyframe bias (#658) | BOTH shipped DiTs load inside the contract; one device-resident on GB10. Caption proj on `vt::MatmulBT` (#1208); LoRA fusion on `vt::Matmul` (#1202), add-back (#1254). #1286 REFUTED (#1317), alloc gated. Render OWED | -| MiniMax-Music3 (`MiniMaxMusic3ForConditionalGeneration`, text-to-MUSIC) | **`ACTIVE`: W0-W7 landed; every stage including the 8.6B LM forward is implemented and gated (#672).** Oracle is the OPEN diffusers PR #14456 `c6da9936` | GGUF arms for 4 components owed. LM gated in a control; HTTP OBSERVED (#852). PARTIAL device arm, Thor sm_110 (#672): 8.6B LM + 2.4B fp32 DiT (§14). Depth 4.45x, wall 2.74x, WAV byte-identical (§16). No reference number | +| MiniMax-Music3 (`MiniMaxMusic3ForConditionalGeneration`, text-to-MUSIC) | **`ACTIVE`: W0-W7 landed; every stage including the 8.6B LM forward is implemented and gated (#672).** Oracle is the OPEN diffusers PR #14456 `c6da9936` | GGUF arms for 4 components owed. LM gated in a control; HTTP OBSERVED (#852). PARTIAL device arm, Thor: 8.6B LM+2.4B fp32 DiT (§14). Depth 4.45x, wall 2.74x (§16); vocoder WINDOW 1.36-1.44x (§18). No reference number | | Command-R / Cohere dense (`CohereForCausalLM`) | Implemented, gate-blocked | ZERO-new-kernel port grounded in vLLM `commandr.py`: weight-only Cohere LayerNorm + GPT-J full-width RoPE + PARALLEL residual + `logit_scale` + tied embeddings, all reuse; compiles, links, self-registers. No SACRED gate yet (real checkpoints HF-gated, ungated ones tiny-random, GPU box disk-full); oracle run-verified at W0. See docs/BENCHMARKS.md | | Phi-1 / Phi-2 dense (`PhiForCausalLM`, parallel residual) | Correctness-complete, speed-pending | Token-exact 16/16 (microsoft/phi-2): 9/16 strict + 7/16 bf16 near-ties (max gap 0.25 nats), 0 forward-divergent; the OLDER Microsoft Phi arch, DISTINCT from Phi-3/Phi-4; ZERO new compute kernel (GPT-J parallel residual, LayerNorm-with-bias, biased qkv/dense, partial NeoX rope 32/80, non-gated NewGELU MLP reusing `vt::GeluTanh`, untied biased lm_head); F16 dtype-aware loader | | MiniCPM dense (`MiniCPMForCausalLM`, three scalars) | Correctness-complete, speed-pending | Token-exact 16/16 (openbmb/MiniCPM-2B-sft-bf16): 10/16 strict + 6/16 bf16 near-ties (max gap 0.0 nats), 0 forward-divergent; first OpenBMB MiniCPM model; ZERO new compute kernel (the Llama/Granite dense forward plus three scalars: scale_emb, scale_depth/sqrt(layers) residual, dim_model_base logit scaling), tied lm_head; `.bin`-only weights converted to safetensors via trusted torch | diff --git a/docs/USAGE.md b/docs/USAGE.md index f8538bb19..708fbdbea 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -2553,6 +2553,37 @@ change while still being the artifact the §16.6 measurement is reproducible fro take the minimum; it prints one fingerprint per process, after its round loop, so a "speedup" that changed the answer cannot be mistaken for one that did not. +To price the **vocoder** the same way, `scripts/music3-vocoder-conv-ab.sh` runs +the whole A/B for you: + +```sh +scripts/music3-vocoder-conv-ab.sh https://github.com/mudler/vllm.cpp +# LENGTHS=20,40,86,172,344 REPEATS=3 ROUNDS=3 JOBS=8 are the knobs +``` + +It clones two trees that differ in `src/vt/cpu/cpu_conv1d_general.cpp` and in +nothing else, builds each in its own directory, and **refuses to time anything +when the two binaries hash the same** — that is the failure that voided this +model's first depth A/B, and equal times are noise where equal binaries are +identity. It then runs the correctness gates on the after arm before reading any +speed number, alternates the arms across a sweep of latent window lengths, and +prints `uptime` on both sides of the sweep. + +The executable it builds, `vllm_music3_vocoder_conv_ab`, can also be run alone +(`--lengths=`, `--repeats=`). It drives `VocoderDecode` — the same call +`vocoder.decode_window` brackets — at the shipped vocoder geometry with +synthetic weights, so it prices that stage without a checkpoint and makes no +claim about audio. It prints one waveform fingerprint per length, which is how +two arms are shown to agree BIT FOR BIT rather than closely. `ctest` never runs +it (#1334). + +**What it times is the WINDOW, not the convolution.** The ratio it prints covers +everything `VocoderDecode` does — `vt::Conv1d`, `vt::ConvTranspose1d`, the +alias-free activations, the strided downsamples, and the threadpool and +allocation around all of them. A kernel-level figure for `vt::Conv1d` alone is +several times larger than the window figure at the same build and thread count, +so the two are not interchangeable and this tool only ever reports the second. + **Measured, so expectations are calibrated rather than hoped for.** On a Jetson Thor (sm_110, 14 cores) the device arm was *slower* on a two-frame request (846.6 s vs 835.1 s) and 5.4 % faster on a ten-frame one (1430.4 s vs 1512.1 s). diff --git a/scripts/music3-vocoder-conv-ab.sh b/scripts/music3-vocoder-conv-ab.sh new file mode 100755 index 000000000..ab41ba847 --- /dev/null +++ b/scripts/music3-vocoder-conv-ab.sh @@ -0,0 +1,103 @@ +#!/usr/bin/env bash +# Two-tree A/B for MiniMax-Music3's vocoder decode window (#672, #1334). +# +# THE SHAPE IS THE POINT. Spec §18.8 binds this row to arms that are two SOURCE +# TREES in two BUILD DIRECTORIES whose binaries have DIFFERENT sha256, because +# §16.6a was voided when both arms turned out to be the same binary and the tell +# was identical call counts rather than equal times. This script therefore +# HARD-FAILS when the two harness binaries hash the same, before it times +# anything. +# +# It builds from a clone rather than from a working tree so that the arms cannot +# share an object cache, and it builds in /tmp rather than on any network mount. +# It is CPU-only: the change under test is a host kernel. +# +# usage: music3-vocoder-conv-ab.sh [workdir] +# +# after-ref the ref carrying the change +# before-ref the ref to take `src/vt/cpu/cpu_conv1d_general.cpp` FROM. Only +# that file is taken, so the two trees differ in the kernel and in +# nothing else — the harness, the config and the build flags are +# byte-identical across the arms. +set -euo pipefail + +url=${1:?git url} +after=${2:?after ref} +before=${3:?before ref} +work=${4:-/tmp/music3-vocoder-ab} +lengths=${LENGTHS:-20,40,86,172,344} +repeats=${REPEATS:-3} +rounds=${ROUNDS:-3} +jobs=${JOBS:-8} +kernel=src/vt/cpu/cpu_conv1d_general.cpp + +say() { printf '\n=== %s\n' "$*"; } + +say "BOX" +uname -a +nproc +grep -m1 'model name\|Model' /proc/cpuinfo || true +echo "uptime(before-build): $(uptime)" + +rm -rf "$work" +mkdir -p "$work" +say "CLONE $after" +git clone --quiet "$url" "$work/after" +git -C "$work/after" checkout --quiet "$after" +after_sha=$(git -C "$work/after" rev-parse HEAD) +cp -a "$work/after" "$work/before" +git -C "$work/before" checkout --quiet "$before" -- "$kernel" +before_sha=$(git -C "$work/before" rev-parse HEAD) + +say "THE ONLY SOURCE DIFFERENCE BETWEEN THE ARMS" +diff -rq "$work/before/src" "$work/after/src" || true +echo "kernel sha256:" +sha256sum "$work/before/$kernel" "$work/after/$kernel" +if [[ "$(sha256sum <"$work/before/$kernel" | cut -d' ' -f1)" == \ + "$(sha256sum <"$work/after/$kernel" | cut -d' ' -f1)" ]]; then + echo "FATAL: the two trees carry the SAME kernel; there is no A/B here" >&2 + exit 4 +fi + +for arm in before after; do + say "BUILD $arm" + cmake -S "$work/$arm" -B "$work/build-$arm" -G Ninja \ + -DCMAKE_BUILD_TYPE=Release -DVLLM_CPP_BUILD_TESTS=ON \ + -DVLLM_CPP_CUDA=OFF -DVLLM_CPP_TRITON=OFF >"$work/cfg-$arm.log" 2>&1 + ninja -C "$work/build-$arm" -j "$jobs" vllm_music3_vocoder_conv_ab \ + >"$work/build-$arm.log" 2>&1 + echo "built $arm: rc=$?" +done + +say "BINARY IDENTITY — the assertion that voided spec 16.6a" +sha256sum "$work/build-before/vllm_music3_vocoder_conv_ab" \ + "$work/build-after/vllm_music3_vocoder_conv_ab" +a=$(sha256sum <"$work/build-before/vllm_music3_vocoder_conv_ab" | cut -d' ' -f1) +b=$(sha256sum <"$work/build-after/vllm_music3_vocoder_conv_ab" | cut -d' ' -f1) +if [[ "$a" == "$b" ]]; then + echo "FATAL: both arms are the SAME BINARY; no measurement is possible" >&2 + exit 5 +fi + +say "CORRECTNESS on the AFTER arm, before any speed number is read" +ninja -C "$work/build-after" -j "$jobs" test_ops_conv1d_general test_host_parallel \ + test_vocoder1d test_bigvgan >>"$work/build-after.log" 2>&1 +for t in test_ops_conv1d_general test_host_parallel test_vocoder1d test_bigvgan; do + echo "--- $t" + "$work/build-after/tests/$t" 2>&1 | grep -E 'test cases:|assertions:|Status:|SKIP' || true + echo "rc=${PIPESTATUS[0]}" +done + +say "SWEEP — arms ALTERNATED, $rounds rounds, best-of-$repeats per point" +echo "uptime(before-sweep): $(uptime)" +for round in $(seq 1 "$rounds"); do + for arm in before after; do + echo "--- round $round arm $arm" + "$work/build-$arm/vllm_music3_vocoder_conv_ab" --lengths="$lengths" --repeats="$repeats" + done +done +echo "uptime(after-sweep): $(uptime)" + +say "PROVENANCE" +echo "after ref=$after sha=$after_sha" +echo "before ref=$before sha=$before_sha (kernel only)" diff --git a/src/vt/cpu/cpu_conv1d_general.cpp b/src/vt/cpu/cpu_conv1d_general.cpp index 9beca6c78..fae70967a 100644 --- a/src/vt/cpu/cpu_conv1d_general.cpp +++ b/src/vt/cpu/cpu_conv1d_general.cpp @@ -65,14 +65,60 @@ void ForOutputRows(int64_t rows, int64_t work_per_row, ParallelForRows(CurrentThreadpool(), rows, body); } +// How many OUTPUT POSITIONS share one sweep of the (ic, k) weights. It is not a +// blocking heuristic and it is not tunable at run time: it is the number of +// INDEPENDENT f64 accumulator chains the kernel offers the machine, and the +// whole of #1334 is that the shipped loop offered exactly one. 32 doubles is +// four AVX-512 registers or sixteen NEON ones, and it measured best of {8, 16, +// 32, 64} at the vocoder's shapes on both -O2 and -O3. +constexpr int64_t kConv1dPosTile = 32; + +// The fixed width the in-range part of a tile is chunked into when the tile is +// not whole. GCC's -O2 vector cost model is `very-cheap` and takes only a loop +// whose trip count is a known multiple of the vector width, so a runtime bound +// is the difference between 1.1x and 5.2x on the SAME source. Release is -O3 +// and CI builds it, but scripts/dgx-bringup.sh builds RelWithDebInfo, and a +// kernel that is fast only under the flag the next measurement happens to pick +// is a kernel nobody can compare. +constexpr int64_t kConv1dChunk = 8; + // out[n, oc, t] = bias[oc] + Sum_{ic,k} x[n, g*Cin/g + ic, t*stride - padding + // k*dilation] * w[oc, ic, k], taps outside [0, Lin) SKIPPED (zero padding). // -// The accumulator is seeded with the bias and walked in (ic ascending, k -// ascending) order — vocoder1d.cpp:90-100 @ 8fa405bb7, verbatim. Every existing -// caller passes padding == 0 (the vocoder pads explicitly through -// `vocoder1d::Pad1d`, which can also replicate); the skip below is therefore -// unreachable on those shapes and exists for torch parity. +// THE ARITHMETIC IS THE ONE vocoder1d.cpp:90-100 @ 8fa405bb7 PERFORMED, AND +// THE ORDER IS TOO. What changed (#1334) is only which cells are in flight at +// once. The shipped form seeded one accumulator with the bias and swept +// (ic ascending, k ascending) into it, which for a MiniMax-Music3 residual unit +// is in_per_group * kernel = 384 * 7 = 2688 STRICTLY DEPENDENT f64 additions per +// output element: no instruction-level parallelism, no vectorisation at any +// width, and a measured ~2.8-3.0 cycles per multiply-accumulate, which is what +// an `fadd` latency of 3 predicts and nothing else does. +// +// This form holds kConv1dPosTile accumulators, one per output position, and +// hoists the (ic, k) sweep outside them. Fix ANY single output cell and read +// what it receives, in order: the bias, then (ic=0,k=0), (ic=0,k=1), ... — the +// identical sequence of IEEE-754 double additions of the identical double +// products, in the identical order. The additions are INTERLEAVED across +// independent cells rather than serialised into one. That is a scheduling +// change and not an arithmetic one, so `memcmp` equality with the pre-change +// loop, with the CUDA provider, and with all four consumers' goldens survives BY +// CONSTRUCTION rather than within a tolerance. +// +// The zero-padding skip moves from a per-position test to a CLAMP on the tile, +// which is the same set: for a fixed k the positions with 0 <= t*stride - pad + +// k*dilation < Lin form one contiguous run of t, so the (t, ic, k) triples this +// skips are exactly the triples the shipped loop skipped. Every existing caller +// passes padding == 0, so the clamp's edges are reached only by the op's torch +// arm — which is why tests/vt/test_ops_conv1d_general.cpp gates them and why +// this row added the tile-geometry cases that do. +// +// stride > 1 keeps the shipped gather (`acc[i] += x[base + i*stride] * w`), +// because a strided load does not vectorise and a gather instruction is slower +// than the scalar it replaces here. The MiniMax-Music3 vocoder's Conv1d calls +// are all stride 1; the strided caller is the alias-free DOWNSAMPLE in +// vocoder1d::AliasFreeActivation1d::Apply, which is depthwise (in_per_group == +// 1) and therefore has a chain one tap deep — it is not the shape this is +// about. .agents/specs/minimax-music3.md §18.4. void Conv1dKernel(Queue&, Tensor& out, const Tensor& x, const Tensor& w, const Tensor* bias, const Conv1dArgs& args) { const int64_t batch = x.shape[0], in_channels = x.shape[1], in_len = x.shape[2]; @@ -88,29 +134,63 @@ void Conv1dKernel(Queue&, Tensor& out, const Tensor& x, const Tensor& w, const T const int64_t rows = batch * out_channels; ForOutputRows(rows, length * in_per_group * kernel, [&](int64_t r0, int64_t r1) { + alignas(64) double acc[kConv1dPosTile]; for (int64_t r = r0; r < r1; ++r) { const int64_t n = r / out_channels; const int64_t oc = r - n * out_channels; const int64_t g = oc / out_per_group; const float* xn = xp + n * in_channels * in_len; float* on = op + (n * out_channels + oc) * length; - for (int64_t t = 0; t < length; ++t) { - double acc = bp != nullptr ? bp[oc] : 0.0; + const double seed = bp != nullptr ? bp[oc] : 0.0; + for (int64_t t0 = 0; t0 < length; t0 += kConv1dPosTile) { + const int64_t tile = std::min(kConv1dPosTile, length - t0); + for (int64_t i = 0; i < tile; ++i) acc[i] = seed; for (int64_t ic = 0; ic < in_per_group; ++ic) { - const int64_t src_c = g * in_per_group + ic; + const float* xc = xn + (g * in_per_group + ic) * in_len; + const float* wc = wp + (oc * in_per_group + ic) * kernel; for (int64_t k = 0; k < kernel; ++k) { - const int64_t pos = t * stride - pad + k * dilation; - if (pos < 0 || pos >= in_len) continue; - acc += static_cast(xn[src_c * in_len + pos]) * - static_cast(wp[(oc * in_per_group + ic) * kernel + k]); + const double wv = static_cast(wc[k]); + // Position of tile slot i is `base + i * stride`; it contributes iff + // that lies in [0, in_len), which bounds i to [lo, hi). + const int64_t base = t0 * stride - pad + k * dilation; + int64_t lo = base < 0 ? (-base + stride - 1) / stride : 0; + const int64_t room = in_len - base; + int64_t hi = room <= 0 ? 0 : std::min(tile, (room + stride - 1) / stride); + if (lo >= hi) continue; + // Formed only after the clamp, so it never points before `xc`. + const float* xs = xc + base + lo * stride; + double* ap = acc + lo; + const int64_t span = hi - lo; + if (stride != 1) { + for (int64_t i = 0; i < span; ++i) + ap[i] += static_cast(xs[i * stride]) * wv; + } else if (span == kConv1dPosTile) { + // The whole-tile case, and the only loop here with a compile-time + // trip count. It is what the vocoder's own shapes take. + for (int64_t i = 0; i < kConv1dPosTile; ++i) + ap[i] += static_cast(xs[i]) * wv; + } else { + int64_t i = 0; + for (; i + kConv1dChunk <= span; i += kConv1dChunk) { + double* a8 = ap + i; + const float* x8 = xs + i; + for (int64_t j = 0; j < kConv1dChunk; ++j) + a8[j] += static_cast(x8[j]) * wv; + } + for (; i < span; ++i) ap[i] += static_cast(xs[i]) * wv; + } } } - on[t] = static_cast(acc); + for (int64_t i = 0; i < tile; ++i) on[t0 + i] = static_cast(acc[i]); } } }); } +// Fixed tap width for the contiguous (dilation == 1) scatter; 4 divides every +// kernel the four consumers run (4, 8, 12, 16) with no tail. +constexpr int64_t kConvTranspose1dTapChunk = 4; + // torch.nn.functional.conv_transpose1d. Weight is [Cin, Cout/groups, K]. // // The SCATTER form, verbatim from vocoder1d.cpp:136-158 @ 8fa405bb7: one @@ -151,7 +231,26 @@ void ConvTranspose1dKernel(Queue&, Tensor& out, const Tensor& x, const Tensor& w const double value = xn[ic * in_len + t]; if (value == 0.0) continue; double* dst = acc.data() + t * stride; - for (int64_t k = 0; k < kernel; ++k) dst[k * dilation] += value * static_cast(wc[k]); + // The taps of ONE input value land in `kernel` DISTINCT cells, so this + // scatter already has instruction-level parallelism and chunking it + // reorders nothing: no cell receives two of these adds. What the chunk + // buys is the compile-time trip count -O2's `very-cheap` vector model + // needs (2.7-2.9x there on the three kernels of 8 taps or more; -O3 + // already finds it, which is why this op is ~6 % of the chain's wall and + // Conv1d is ~94 %). §18.5. + if (dilation == 1) { + int64_t k = 0; + for (; k + kConvTranspose1dTapChunk <= kernel; k += kConvTranspose1dTapChunk) { + double* dk = dst + k; + const float* wk = wc + k; + for (int64_t j = 0; j < kConvTranspose1dTapChunk; ++j) + dk[j] += value * static_cast(wk[j]); + } + for (; k < kernel; ++k) dst[k] += value * static_cast(wc[k]); + } else { + for (int64_t k = 0; k < kernel; ++k) + dst[k * dilation] += value * static_cast(wc[k]); + } } } float* on = op + (n * out_channels + dst_c) * length; diff --git a/tests/vt/test_ops_conv1d_general.cpp b/tests/vt/test_ops_conv1d_general.cpp index 62e173cf2..057b9a0e7 100644 --- a/tests/vt/test_ops_conv1d_general.cpp +++ b/tests/vt/test_ops_conv1d_general.cpp @@ -122,7 +122,8 @@ void RequireBitIdentical(const std::vector& got, const std::vector std::vector SerialConv1d(const std::vector& in, int64_t batch, int64_t in_channels, int64_t in_len, const std::vector& weight, const std::vector* bias, int64_t out_channels, - int64_t kernel, const Conv1dArgs& a, int64_t length) { + int64_t kernel, const Conv1dArgs& a, int64_t length, + bool reverse_ic = false, bool reverse_k = false) { const int64_t in_per_group = in_channels / a.groups; const int64_t out_per_group = out_channels / a.groups; std::vector out(static_cast(batch * out_channels * length), 0.0F); @@ -131,9 +132,11 @@ std::vector SerialConv1d(const std::vector& in, int64_t batch, int const int64_t g = oc / out_per_group; for (int64_t t = 0; t < length; ++t) { double acc = bias != nullptr ? (*bias)[static_cast(oc)] : 0.0; - for (int64_t ic = 0; ic < in_per_group; ++ic) { + for (int64_t step = 0; step < in_per_group; ++step) { + const int64_t ic = reverse_ic ? in_per_group - 1 - step : step; const int64_t src_c = g * in_per_group + ic; - for (int64_t k = 0; k < kernel; ++k) { + for (int64_t tap = 0; tap < kernel; ++tap) { + const int64_t k = reverse_k ? kernel - 1 - tap : tap; const int64_t pos = t * a.stride - a.padding + k * a.dilation; if (pos < 0 || pos >= in_len) continue; acc += static_cast( @@ -149,10 +152,22 @@ std::vector SerialConv1d(const std::vector& in, int64_t batch, int return out; } -// `reverse_ic` walks the input channels DESCENDING instead of ascending. It is -// not a mode the op has; it is the ORDER MUTATION the cancellation case uses to -// prove it has teeth before any agreement is believed. Same multiset of -// products into every destination cell, different sequence of additions. +// `reverse_ic` and `reverse_k` walk the input channels / the taps DESCENDING +// instead of ascending. Neither is a mode the op has; each is the ORDER MUTATION +// a cancellation case uses to prove it has teeth before any agreement is +// believed. Same multiset of products into every destination cell, different +// sequence of additions. +// +// THERE ARE TWO AXES AND EACH NEEDS ITS OWN CASE, which is a finding rather than +// a symmetry (#1334, review 2026-08-19). The sweep the CPU kernel hoists out of +// its accumulators is `(ic ascending, k ascending)`, and reversing EITHER is a +// genuine reassociation. Data engineered to cancel across `ic` does not bite +// along `k`: with the kernel's `k` loop reversed and nothing else changed, +// `test_ops_conv1d_general` 9/375, `test_host_parallel` 8/877, +// `test_vocoder1d` 10/58 and `test_bigvgan` 6/65 all reported `SUCCESS!` — +// measured on this tree, binary sha256 `c4bb0e76...` against the baseline +// `760061c5...`, so the mutation was in the binary that ran. A case per axis is +// therefore what the guarantee costs. std::vector SerialConvTranspose1d(const std::vector& in, int64_t batch, int64_t in_channels, int64_t in_len, const std::vector& weight, @@ -228,6 +243,21 @@ const FwdCase kFwdCases[] = { {"batch3", 3, 24, 96, 24, 3, {1, 1, 1, 1}, false}, // Kernel longer than the input once padded in. {"k-gt-l", 1, 8, 5, 8, 9, {1, 4, 1, 1}, true}, + // --- THE TILE GEOMETRY (#1334) ----------------------------------------- + // The CPU forward kernel walks output positions in tiles of 32, so the tile + // boundary is an edge the cases above only meet by coincidence. These pick + // lengths AGAINST it: exactly one tile, one tile plus a 1-wide tail, a + // length below one tile, and a length whose left clamp falls in the first + // tile while its right clamp falls in a partial LAST tile — the arrangement + // a per-position skip cannot get wrong and a per-tile clamp can. + {"tile-exact-32", 1, 24, 38, 24, 7, {1, 0, 1, 1}, true}, + {"tile-plus-one", 1, 24, 39, 24, 7, {1, 0, 1, 1}, false}, + {"sub-tile-padded", 1, 16, 20, 16, 7, {1, 3, 1, 1}, true}, + {"multi-tile-padded", 1, 12, 100, 12, 9, {1, 4, 1, 1}, true}, + {"tile-boundary-dilated", 1, 16, 96, 16, 5, {1, 6, 4, 1}, false}, + // The strided arm keeps the shipped gather, and its clamp is the one with a + // ceiling division in it: `lo = ceil(-base/stride)`, `hi = ceil(room/stride)`. + {"tile-strided-padded", 1, 16, 200, 16, 5, {3, 4, 2, 1}, true}, }; struct BwdCase { @@ -429,6 +459,165 @@ TEST_CASE("vt::ConvTranspose1d reproduces the host loop's ZERO-SKIP exactly") { for (const float v : want) CHECK(std::signbit(v) == false); } +TEST_CASE("vt::Conv1d CPU holds its sweep ORDER under cancellation, across tile boundaries") { + // THE case for #1334 along the `ic` AXIS, and a hole this file had before it. + // The `k` axis is the case below, and the two are not interchangeable: this + // one pairs input CHANNELS and its TEETH check reverses `ic`, so a `k` sweep + // reversed in the kernel walks straight through it. The cancellation case at + // the bottom compares Conv1d CPU-vs-CUDA only, so on a CPU-only build + // NOTHING here held the forward sweep's ORDER against the pre-op host loop — + // every other forward assertion runs on well-scaled data, where an f64 + // accumulator stored through an f32 hides a reduction-order change completely + // (measured, not assumed: tests/vllm/models/test_host_parallel.cpp). + // + // That is exactly the guarantee the tiled kernel turns on. It interleaves the + // (ic, k) sweep across 32 independent accumulators instead of serialising it + // into one, which must leave the order each individual cell sees UNCHANGED. On + // benign data a kernel that got that wrong would still pass everything above. + // + // The shape is chosen against the tile: length 197 is six whole tiles plus a + // 5-wide tail, padding 3 puts the left clamp inside tile 0 and the right clamp + // inside that tail, and dilation 2 makes the two clamps land at different taps. + const int64_t cin = 32, lin = 205, cout = 24, kernel = 8; + const float kBig = 1099511627776.0F; // 2^40 + std::vector in = Spread(static_cast(cin * lin), 0x3C3Cu); + std::vector w = Spread(static_cast(cout * cin * kernel), 0xC3C3u); + for (int64_t t = 0; t < lin; ++t) { + in[static_cast(0 * lin + t)] = kBig; + in[static_cast(1 * lin + t)] = -kBig; + } + // Channels 0 and 1 share a weight row, so the sequential (ic ascending) sweep + // cancels them at once and keeps the small remainder exactly; any other order + // carries 2^40 through the accumulator and quantises at ~1.2e-4. + for (int64_t oc = 0; oc < cout; ++oc) { + for (int64_t k = 0; k < kernel; ++k) { + w[static_cast((oc * cin + 1) * kernel + k)] = + w[static_cast((oc * cin + 0) * kernel + k)]; + } + } + + Conv1dArgs args; + args.stride = 1; + args.padding = 3; + args.dilation = 2; + const int64_t lout = vt::Conv1dOutLength(lin, kernel, args); + REQUIRE(lout == 197); + const std::vector want = + SerialConv1d(in, 1, cin, lin, w, nullptr, cout, kernel, args, lout); + const std::vector got = RunFwd(Cpu(), in, {1, cin, lin}, w, {cout, cin, kernel}, nullptr, + {1, cout, lout}, args); + RequireBitIdentical(got, want, "Conv1d cancellation cpu-vs-oracle"); + + // TEETH, before the agreement above is believed. Reversing the input-channel + // sweep is a genuine reassociation — the same multiset of products into every + // cell, added in the opposite order. If this reads 0 the cancellation has + // stopped biting and the equality above is vacuous, which is the one state a + // passing suite cannot otherwise report. + { + const std::vector other = + SerialConv1d(in, 1, cin, lin, w, nullptr, cout, kernel, args, lout, /*reverse_ic=*/true); + size_t differing = 0; + for (size_t i = 0; i < other.size(); ++i) { + if (std::memcmp(&other[i], &want[i], sizeof(float)) != 0) ++differing; + } + INFO("order sensitivity: " << differing << " of " << want.size() + << " cells change when the ic sweep is reversed"); + CHECK(differing > 0); + } + + if (!HasCuda()) { + std::printf("[SKIP] no CUDA backend: Conv1d tiled-cancellation CPU-vs-CUDA NOT exercised\n"); + return; + } + const std::vector cuda = RunFwd(Device{DeviceType::kCUDA, 0}, in, {1, cin, lin}, w, + {cout, cin, kernel}, nullptr, {1, cout, lout}, args); + RequireBitIdentical(cuda, got, "Conv1d cancellation cuda-vs-cpu across tiles"); +} + +TEST_CASE("vt::Conv1d CPU holds its TAP order under cancellation, across tile boundaries") { + // The SECOND axis, and the reason it is a separate case (#1334, review + // 2026-08-19). The kernel hoists `(ic ascending, k ascending)` out of 32 + // accumulators, so reversing EITHER loop reassociates every cell — but the + // case above is engineered along `ic` alone, and its own teeth check reverses + // `ic` alone. Reversing only the kernel's `k` loop left the whole gate green: + // 9/375, 8/877, 10/58, 6/65, every one `SUCCESS!`. A guarantee gated on one of + // its two axes is gated on neither, so this case pairs TAPS instead. + // + // The construction, and why it bites. Input channel 0 is CONSTANT at 1.0, so + // the taps of that channel all read the same value whatever `dilation` does + // with their positions. Its first two taps are +2^40 and -2^40 and its + // remaining six are O(1), and every other channel is O(1) throughout. Sweeping + // `k` ascending cancels 2^40 against 2^40 at once and then adds the O(1) + // remainder into an accumulator of magnitude ~0, exactly. Sweeping it + // descending accumulates that remainder FIRST and then puts +-2^40 through it, + // which quantises it at 2^-12 ~ 2.4e-4 — five orders of magnitude above the + // f32 store's ~2e-6 ULP at this result's scale, so the difference cannot hide + // in the round-trip through `float` the way a well-scaled reordering does. + // + // The shape is the one the `ic` case uses, for the same reason: `lout == 197` + // is six whole 32-wide tiles plus a 5-wide tail, `padding == 3` puts the left + // clamp inside tile 0, and `dilation == 2` lands the two big taps at different + // positions. It also drives the clamp THROUGH the cancelling pair: at t=0..1 + // both big taps are out of range, at t=2 only the negative one is in range, + // and from t=3 on both are — so a clamp that keeps the wrong tap produces 2^40 + // rather than a rounding difference. + const int64_t cin = 4, lin = 205, cout = 8, kernel = 8; + const float kBig = 1099511627776.0F; // 2^40 + std::vector in(static_cast(cin * lin)); + std::vector w(static_cast(cout * cin * kernel)); + // O(1) and deterministic: the same LCG `Spread` uses, mapped into [0.5, 1.5) + // instead of across six decades. The magnitude is the point — a small term + // that is itself huge is not a remainder. + uint32_t s = 0x5EEDu; + const auto next = [&s]() { + s = s * 1664525U + 1013904223U; + return 0.5F + static_cast(s >> 8) / 33554432.0F; // [0.5, 1.5) + }; + for (float& v : in) v = next(); + for (float& v : w) v = next(); + for (int64_t t = 0; t < lin; ++t) in[static_cast(0 * lin + t)] = 1.0F; + for (int64_t oc = 0; oc < cout; ++oc) { + w[static_cast((oc * cin + 0) * kernel + 0)] = kBig; + w[static_cast((oc * cin + 0) * kernel + 1)] = -kBig; + } + + Conv1dArgs args; + args.stride = 1; + args.padding = 3; + args.dilation = 2; + const int64_t lout = vt::Conv1dOutLength(lin, kernel, args); + REQUIRE(lout == 197); + const std::vector want = + SerialConv1d(in, 1, cin, lin, w, nullptr, cout, kernel, args, lout); + const std::vector got = RunFwd(Cpu(), in, {1, cin, lin}, w, {cout, cin, kernel}, nullptr, + {1, cout, lout}, args); + RequireBitIdentical(got, want, "Conv1d tap-order cancellation cpu-vs-oracle"); + + // TEETH, on THIS axis, before the agreement above is believed. Reversing the + // tap sweep is the mutation this case exists to detect; if it changes nothing + // the equality above is vacuous. + { + const std::vector other = + SerialConv1d(in, 1, cin, lin, w, nullptr, cout, kernel, args, lout, + /*reverse_ic=*/false, /*reverse_k=*/true); + size_t differing = 0; + for (size_t i = 0; i < other.size(); ++i) { + if (std::memcmp(&other[i], &want[i], sizeof(float)) != 0) ++differing; + } + INFO("tap order sensitivity: " << differing << " of " << want.size() + << " cells change when the k sweep is reversed"); + CHECK(differing > 0); + } + + if (!HasCuda()) { + std::printf("[SKIP] no CUDA backend: Conv1d tap-order cancellation CPU-vs-CUDA NOT exercised\n"); + return; + } + const std::vector cuda = RunFwd(Device{DeviceType::kCUDA, 0}, in, {1, cin, lin}, w, + {cout, cin, kernel}, nullptr, {1, cout, lout}, args); + RequireBitIdentical(cuda, got, "Conv1d tap-order cancellation cuda-vs-cpu across tiles"); +} + TEST_CASE("vt conv1d ops REFUSE a narrow dtype by name rather than widening it") { // f16/bf16 arms are not implemented. AGENTS.md requires an unimplemented arm // to be refused with a message naming the missing piece, not to be silently diff --git a/tools/bench/music3_vocoder_conv_ab.cpp b/tools/bench/music3_vocoder_conv_ab.cpp new file mode 100644 index 000000000..9fd0e8e6d --- /dev/null +++ b/tools/bench/music3_vocoder_conv_ab.cpp @@ -0,0 +1,190 @@ +// A/B driver for MiniMax-Music3's vocoder decode window (#672, #1334). +// +// WHAT IT MEASURES, AND WHAT IT IS NOT. It calls `music3::VocoderDecode` — the +// production entry point `Music3DecodeChunks` calls inside the +// `vocoder.decode_window` profile bracket (minimax_music3_speech.cpp:397-398) — +// at the shipped `MiniMaxMusic3VocoderConfig` geometry, for a sweep of latent +// window lengths. It is therefore that bucket's computation and not a +// reconstruction of it. It is NOT an end-to-end synthesis: the weights are +// synthetic, so nothing here is a claim about audio, and it takes no GPU clock +// window (.agents/benchmarking.md), so nothing here is a per-kernel or +// cross-box figure. +// +// WHY SYNTHETIC WEIGHTS ARE HONEST HERE. The conv kernels' cost does not depend +// on the weight VALUES, with one exception that is handled rather than ignored: +// `vt::ConvTranspose1d` skips an input that compares equal to 0.0, so a +// degenerate all-zero input would measure a path the real one never takes. The +// generator below emits no exact zero, and both arms are seeded identically, so +// the two arms execute the same number of the same operations. +// +// HOW THE ARMS ARE SEPARATED. There is no `#if` here and there is no flag. The +// two arms are two SOURCE TREES that differ in +// `src/vt/cpu/cpu_conv1d_general.cpp` and nothing else, built in two build +// directories, and the harness prints a checksum of the waveform so the run can +// assert that the arms agree BIT FOR BIT while their binaries differ. Equal +// times are noise; equal binaries are identity, and that is what voided the +// first attempt at spec §16.6a. +// +// CI NEVER RUNS THIS. It is an executable rather than a registered test, it +// allocates hundreds of megabytes of synthetic weights, and it spends tens of +// seconds per sweep. It is compiled by the ordinary build so that the one file a +// reader has to compile to reproduce the measurement cannot rot silently. +#include +#include +#include +#include +#include +#include +#include +#include + +#include "vllm/model_executor/models/minimax_music3_acoustic.h" +#include "vllm/model_executor/models/minimax_music3_loader.h" + +namespace { + +using vllm::MiniMaxMusic3VocoderConfig; +using vllm::models::music3::VocoderBlockWeights; +using vllm::models::music3::VocoderResidualUnitWeights; +using vllm::models::music3::VocoderWeights; + +// A plain LCG, identical on every box and every arm. Values are spread over a +// few decades and NEVER exactly zero, so the transposed op's zero-skip is off +// the path and both arms perform the same operation count. +struct Lcg { + uint32_t s; + float Next() { + s = s * 1664525U + 1013904223U; + const double mantissa = static_cast(s >> 8) / 16777216.0 - 0.5; + const int exponent = static_cast((s >> 4) & 0x7U) - 6; + double v = mantissa * std::ldexp(1.0, exponent); + if (v == 0.0) v = 1.0 / 4096.0; + return static_cast(v); + } +}; + +std::vector Fill(Lcg& rng, size_t n) { + std::vector v(n); + for (size_t i = 0; i < n; ++i) v[i] = rng.Next(); + return v; +} + +// The shipped geometry, read off the config rather than restated: decoder_hidden +// halves at each of the four upsampling ratios, exactly as `VocoderDecode` walks +// it (minimax_music3_acoustic.cpp::VocoderDecode). +VocoderWeights SyntheticWeights(const MiniMaxMusic3VocoderConfig& config, Lcg& rng) { + VocoderWeights w; + const int64_t stream = config.stream_channels(); + w.dec_in_proj_weight = Fill(rng, static_cast(config.decoder_input_dim * stream)); + w.dec_in_proj_bias = Fill(rng, static_cast(config.decoder_input_dim)); + w.conv_in_weight = + Fill(rng, static_cast(config.decoder_hidden_dim * config.decoder_input_dim * 7)); + w.conv_in_bias = Fill(rng, static_cast(config.decoder_hidden_dim)); + int64_t last_output = config.decoder_hidden_dim; + for (size_t index = 0; index < config.upsampling_ratios.size(); ++index) { + const int64_t input_dim = config.decoder_hidden_dim >> index; + const int64_t output_dim = config.decoder_hidden_dim >> (index + 1); + const int64_t stride = config.upsampling_ratios[index]; + last_output = output_dim; + VocoderBlockWeights block; + block.snake1_alpha = Fill(rng, static_cast(input_dim)); + block.conv_t1_weight = Fill(rng, static_cast(input_dim * output_dim * 2 * stride)); + block.conv_t1_bias = Fill(rng, static_cast(output_dim)); + for (int unit = 0; unit < vllm::models::music3::kVocoderResidualUnits; ++unit) { + VocoderResidualUnitWeights res; + res.snake1_alpha = Fill(rng, static_cast(output_dim)); + res.conv1_weight = Fill(rng, static_cast(output_dim * output_dim * 7)); + res.conv1_bias = Fill(rng, static_cast(output_dim)); + res.snake2_alpha = Fill(rng, static_cast(output_dim)); + res.conv2_weight = Fill(rng, static_cast(output_dim * output_dim)); + res.conv2_bias = Fill(rng, static_cast(output_dim)); + block.res_units.push_back(std::move(res)); + } + w.blocks.push_back(std::move(block)); + } + w.snake_out_alpha = Fill(rng, static_cast(last_output)); + w.conv_out_weight = Fill(rng, static_cast(last_output * 7)); + w.conv_out_bias = Fill(rng, 1); + return w; +} + +// A bitwise fingerprint of the waveform: FNV-1a over the raw float bytes. A sum +// would let two different waveforms agree; this cannot, and it is what lets the +// two arms be compared for BIT identity rather than for closeness. +uint64_t Fingerprint(const std::vector& v) { + uint64_t h = 1469598103934665603ULL; + const unsigned char* p = reinterpret_cast(v.data()); + for (size_t i = 0; i < v.size() * sizeof(float); ++i) { + h ^= static_cast(p[i]); + h *= 1099511628211ULL; + } + return h; +} + +} // namespace + +int main(int argc, char** argv) { + std::vector lengths; + int repeats = 3; + for (int i = 1; i < argc; ++i) { + const std::string a = argv[i]; + if (a.rfind("--lengths=", 0) == 0) { + std::string list = a.substr(10); + size_t pos = 0; + while (pos <= list.size()) { + const size_t comma = list.find(',', pos); + const std::string one = list.substr(pos, comma == std::string::npos ? comma : comma - pos); + if (!one.empty()) lengths.push_back(std::strtoll(one.c_str(), nullptr, 10)); + if (comma == std::string::npos) break; + pos = comma + 1; + } + } else if (a.rfind("--repeats=", 0) == 0) { + repeats = static_cast(std::strtol(a.substr(10).c_str(), nullptr, 10)); + } else { + std::fprintf(stderr, "usage: %s [--lengths=20,40,...] [--repeats=N]\n", argv[0]); + return 2; + } + } + if (lengths.empty()) lengths = {20, 40, 86, 172, 344}; + + const MiniMaxMusic3VocoderConfig config; + Lcg rng{0x51F0C0DEu}; + const VocoderWeights weights = SyntheticWeights(config, rng); + + std::printf("# MiniMax-Music3 vocoder decode-window A/B (#1334)\n"); + std::printf("# latent_channels=%lld decoder_input_dim=%lld decoder_hidden_dim=%lld hop=%lld\n", + static_cast(config.latent_channels), + static_cast(config.decoder_input_dim), + static_cast(config.decoder_hidden_dim), + static_cast(config.hop_length())); + std::printf("%10s %10s %12s %12s %20s\n", "latents", "repeats", "best_s", "s_per_latent", + "fingerprint"); + for (const int64_t length : lengths) { + Lcg lrng{0x1234ABCDu + static_cast(length)}; + const std::vector latents = + Fill(lrng, static_cast(config.latent_channels * length)); + double best = 1e30; + uint64_t fp = 0; + for (int r = 0; r < repeats; ++r) { + int64_t samples = 0; + const auto t0 = std::chrono::steady_clock::now(); + const std::vector wave = + vllm::models::music3::VocoderDecode(latents, length, config, weights, &samples); + const double dt = std::chrono::duration(std::chrono::steady_clock::now() - t0).count(); + if (dt < best) best = dt; + const uint64_t here = Fingerprint(wave); + if (r == 0) { + fp = here; + } else if (here != fp) { + std::fprintf(stderr, "FATAL: the decode is not deterministic within one arm at length %lld\n", + static_cast(length)); + return 3; + } + } + std::printf("%10lld %10d %12.4f %12.6f %#20llx\n", static_cast(length), repeats, + best, best / static_cast(length), + static_cast(fp)); + std::fflush(stdout); + } + return 0; +}