perf(MUSIC3-VOCODER-CONV-SPEED): the vocoder conv stops running one dependent f64 add chain per output cell (#672, #1334) by localai-bot · Pull Request #1356 · mudler/vllm.cpp

localai-bot · 2026-08-19T12:04:23Z

vocoder.decode_window was the largest term in a MiniMax-Music3 run that no row
owned, and spec §16.7 named it before this row existed. This is §18.

Why it was slow

vt::cpu::Conv1dKernel computed each output cell with ONE f64 accumulator swept
over (ic ascending, k ascending). For a Music3 residual unit that is
in_per_group * kernel = 384 * 7 = 2688 strictly dependent f64 additions per
output element: no instruction-level parallelism, no vectorisation at any width.
Its measured rate on one core of the 20-core Zen 5 is 1.76-1.86 GMAC/s.

That rate is quoted without a cycle count, and the cycle count this PR first
carried is withdrawn. It read "~2.8-3.0 cycles per multiply-accumulate on a
5.0 GHz Zen 5 whose fadd latency is 3", which is not self-consistent: 2.0
GMAC/s at 5.0 GHz is 2.5 cycles per MAC, and a strictly dependent chain of
latency-3 fadds cannot exceed 5.0 / 3 = 1.67 GMAC/s at all, so the top of the
band sat above its own ceiling. The instrument is the deeper problem — the x86
host is a KVM guest reporting a nominal 4291.948 MHz with no cpufreq interface
and perf_event_paranoid at 4, so neither its boost clock nor a cycle count is
observable from inside it and every such figure was a conversion through an
assumed clock. What survives is the identity, that a dependent chain of
latency-L adds cannot beat clock / L, and the 2.16x that breaking the chain
bought. §18.2 carries the correction.

Why the CUDA arm also loses is offered as an INFERENCE, not established.
Checked by reading cuda_conv1d_general.cu: one f64 accumulator and one thread
per cell, __dadd_rn/__dmul_rn. Measured: §13.10's per-stage ratios flat at
0.37-0.40x across a 150x span, which rules out a fixed staging overhead and
little else. Inferred, and OWED in §18.9 with the three instruments that would
settle it: that the residual is fp64 THROUGHPUT on Thor's consumer Blackwell. No
counter was read, no fp64:fp32 ratio taken, no CUDA A/B run.

What changed, and why `memcmp` survives by construction

The f64 stays. §13.2 records it as what all four consumers' goldens were taken
with (Music3, MiniMax-H3's audio VAE, LTX-2.5's audio VAE, IndexTTS-2.5) and what
makes the CUDA provider byte-identical to the host. Narrowing to torch's f32
re-gates four shipped models and is left OWED.

The chain is broken without touching the width. The kernel holds 32 f64
accumulators, one per output position, and hoists the (ic, k) sweep outside
them. Fix any single cell and read what it receives, in order: the bias, then
(ic=0,k=0), (ic=0,k=1), ... — the identical sequence of IEEE-754 double
additions of the identical double products, in the identical order. The additions
are INTERLEAVED across independent cells rather than serialised into one. That is
a scheduling change and not an arithmetic one. The zero-padding skip becomes a
CLAMP on the tile, which is the same set of (t, ic, k) triples, because for a
fixed k the in-range positions form one contiguous run.

The constant trip count is load-bearing, not cosmetic: GCC's -O2 cost model is
very-cheap and takes only a loop whose trip count is a known multiple of the
vector width. On the same source a runtime-bounded loop gives 1.1-1.8x at -O2
and 5.1-5.4x at -O3; the whole-tile constant-trip path gives 5.2-5.8x at -O2
and 5.3-6.5x at -O3. Release is -O3 and scripts/dgx-bringup.sh is
RelWithDebInfo, so a kernel fast only under one of them is a kernel nobody can
compare. stride > 1 keeps the shipped gather.

Every one of those six figures is the KERNEL, never the decode window. The
window also runs vt::ConvTranspose1d, the alias-free activations, the strided
downsamples and the pool around them. The review that asked for the distinction
built the real project at -O2 and measured 2.56x / 2.67x on the window
where the kernel gives 5.2-5.8x. §18.4 says so, and docs/BENCHMARKS.md no
longer calls the single-thread 2.157x a kernel number, because it is the window
on one thread.

The gate, and the axis it did not hold

test_ops_conv1d_general's Conv1d cancellation arm compared CPU against CUDA
ONLY, so on a CPU-only build nothing held the forward sweep's ORDER against the
pre-op host loop, along either axis — and every other forward assertion runs on
well-scaled data, where an f64 accumulator stored through an f32 hides a
reduction-order change completely.

The first version of this row closed that along ic and left k open, which a
fresh review caught. The sweep is (ic ascending, k ascending) and reversing
EITHER loop reassociates every cell, but the case pairs input CHANNELS and its
own teeth check reverses ic alone.

	result
GREEN, this row's gate	`test_ops_conv1d_general` 10 cases / 379 assertions, 0 failed, `SUCCESS!` rc 0
M1 reverse the tiled kernel's `ic` sweep	FAILURE, 1 failed (binary `8bfd1eb4`, compile rc 0)
M1 against the BASE gate at `f06b9e93d`	`SUCCESS!`, 8 cases / 347 assertions — the first hole
M2 break ONLY the whole-tile constant-trip path	`test_ops_conv1d_general` 11/375 failed, `test_host_parallel` 20/877 failed
M3 reverse the `k` sweep, against the gate as first written	`SUCCESS!` on ALL FOUR suites — 9/375, 8/877, 10/58, 6/65, rc 0, binary `c4bb0e76` vs baseline `760061c5` — the second hole
M3 against the gate with the tap case	FAILURE, 10 cases / 379 assertions, 1 failed, 1552 of 1576 cells wrong in exactly that case (binary `eb25d992`)
restored	10/379 `SUCCESS!`, binary back to `f84d3a87`; the other three suites never left `efcff589` / `b01c6f57` / `b3ef384b`

M2 is the reachability mutation: it corrupts the fast path and nothing else, and
the four consumers' own entry-point gate reds, so production shapes DO enter the
new path rather than routing around it.

The k hole is OLDER than this row — the pre-change kernel with k reversed is
green too — so it is not a regression this row introduced. It is still this
row's to close, because this row is what turns that order into a load-bearing
guarantee. SerialConv1d grows a reverse_k beside reverse_ic, and a second
cancellation case pairs +2^40 against -2^40 across k inside one input channel
held constant along its length, so its taps read the same value whatever
dilation does with their positions. Ascending k cancels them at once and
keeps the O(1) remainder exactly; descending accumulates that remainder first and
quantises it at 2^-12, five orders above the f32 store's ULP at that scale. The
clamp is driven THROUGH the pair: at t=0..1 both big taps are out of range, at
t=2 only the negative one is, and from t=3 both are.

Speed — two `rc` leases, four binaries, no `ssh` and no file mutex

da3a2f94 on thor:gpu0 (--max-runtime 150m) built both arms INSIDE the lease
from two clones in local /tmp, Release, CPU-only. diff -rq names
src/vt/cpu/cpu_conv1d_general.cpp as the only difference; the recipe hard-fails
when the two binaries hash the same (d90e3912 vs 41ba78d2), which is what
voided the depth row's first Thor pair. Correctness first, on the after arm:
test_ops_conv1d_general, test_host_parallel, test_vocoder1d and
test_bigvgan all SUCCESS! rc 0.

Arms alternated, 3 rounds, best-of-3, DEFAULT (host) arm. Every ratio below is
the DECODE WINDOW. Medians:

latent frames	before	after
20	5.5688	4.0831	1.364x
40	11.0535	7.8186	1.414x
86	23.5149	16.6614	1.411x
172	47.9201	33.6498	1.424x
344	97.4463	67.7083	1.439x

The weakest pair is kept, and it is the size §15.9 priced the device arm at. Flat
over a 17x span of work, so it is a rate change and not a fixed overhead. The
waveform fingerprint is ONE value per length across all six arm-rounds, so the
arms are bit-identical at full scale.

And a second lease split that number. 5b98f95e (--max-runtime 45m), fresh
container, both arms rebuilt into a NEW pair of binaries:

threads	before	after
1	37.0508 s	17.1751 s	2.157x
14, same container, same binaries	5.4845 s	4.0182 s	1.365x

The 14-thread control reproduces 1.364x to three digits on different binaries.
Scaling 1 -> 14 threads is 6.76x before and 4.27x after, of a possible 14x —
neither arm scales and the FASTER one scales worse, the signature of a shared
resource the tiled kernel reaches sooner. The window is worth 2.16x per core;
the threadpool returns 1.37x. That names the next lever as the vocoder's
parallel decomposition (ForOutputRows partitions output channels, so all 14
threads sweep the whole input tensor) rather than as a ceiling, and it settles the
aarch64-vs-x86 question: per core the same source is 2.16x here against a much
larger kernel-level figure on AVX-512, which a 4x narrower f64 vector explains.

Stated plainly rather than implied

No checkpoint was read. The driver uses synthetic weights at the shipped
geometry, so §18.8's staging assertion is NOT APPLICABLE rather than skipped —
there is no path to assert. The e2e pair on the real checkpoint is OWED.
53.6 s is the CUDA arm. §15.2's profile ran VLLM_CPP_VOCODER_DEVICE=cuda,
while the default is cpu (§13.6) and that is the arm this moves. The two are
recorded side by side and never multiplied.
The CUDA-vs-CPU memcmp arm was NOT re-measured. Thor's worker has no
nvcc, so every CPU-vs-CUDA case printed [SKIP]. §18.3's argument says it
must still hold; an argument is not a measurement, and it is OWED.
A correction to §13.10, which recorded that worker as having "no compiler
and no toolchain at all": probed 2026-08-19 it carries gcc, g++, cmake,
ninja, make, python3 and git. For a CPU arm that blocker does not exist.
.agents/issue-index.md is append-only and its MiniMax-Music3: the vocoder is 33.2% of a run, and its conv kernel is one DEPENDENT f64 add chain per output cell — breakable bit-identically #1334 row still carries the
superseded rate sentence. The correction lives in the spec, which is where a
correction belongs; the row is not edited.
The A/B driver is an add_executable where its sibling is an OBJECT
library. Argued in place: scripts/music3-vocoder-conv-ab.sh builds it,
sha256sums both arms, refuses to time equal binaries and then RUNS them.
Every step needs a linked binary, and it is in no install(TARGETS ...).
docs/STATUS.md and the spec's ## Now now move together. The row does
not change lifecycle state, so by the trigger table neither was owed; the pair
is written together and says so.
No parity claim. SGLang-Omni is still gateable = no.

Gate

Re-run on the merged head at origin/main 31f93787c, because a merge can drop
a case silently and this branch merged main five times while the repairs were
written.

scripts/agent-integration.py --base origin/main: 84 gates, 84 ok, 0 FAIL, 0
SKIP — "All gates green". The wrapper still exits 1 on one assertion that is
about this worktree rather than this tree: agent-ready requires exactly one
live pull request whose headRefName equals the LOCAL branch, and the repair
worktree's branch is row/MUSIC3-VOCODER-CONV-SPEED-REVIEW-FIX while the pull
request rides row/MUSIC3-VOCODER-CONV-SPEED, so it finds 0.

ctest on a CPU-only Release build: 559 of 560 passed, 3 skipped by
design. The one red is test_nemotron_h_paged_forward, "No valid attention
backend for device type 0" — a message that entered in 369ea7fd4, which is not
in this branch's own history, and which reproduces on main.
#1371 tracks it. Not repaired
here: it has an owner and a filed gap.

test_check_gate_commands was red on an earlier head of this merge chain, from
ENG-CUDAGRAPH-BREAK becoming runnable in 601b576c6 without re-pinning
RUNNABLE_BASELINE (#1376).
It was reproduced on a clean origin/main worktree at that SHA and is green
again at 31f93787c; recorded because the earlier head of this branch carried
it and a reader comparing runs would otherwise have to guess.

The forge reported this pull request CONFLICTING while
git merge-tree --write-tree returned rc 0 and printed nothing. That is
.agents/issue-index.md's merge=union attribute, which GitHub does not apply
— reproduced with git merge-file over the same three blobs under the default
driver, which exits 1 on that file alone. The last merge materialises the union
merge in a commit, and the pull request reads MERGEABLE after it.

Tracked by #1334, under the
MiniMax-Music3 lane #672.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [Claude Code]

…f64 add chain per output cell (#672, #1334) §16.7 named this row before it existed: the vocoder is 53.6 s of a 161.6 s run (33.2 %) after #1238, against 12.0 % before it, and #1238's own depth A/B measured it at 53.6 s on BOTH legs — it is a FIXED term that grows as a share of every other improvement. §18 states the cause, which is one cause wearing two mechanisms. `vt::cpu::Conv1dKernel` computes each output cell with ONE f64 accumulator swept over (ic ascending, k ascending). For a MiniMax-Music3 residual unit that is 384 * 7 = 2688 strictly dependent f64 additions per output element: no instruction-level parallelism, no vectorisation at any width, and a measured 1.7-2.0 GMAC/s per core, ~2.8-3.0 cycles per multiply-accumulate on a 5.0 GHz Zen 5 whose `fadd` latency is 3. The CUDA provider loses (§15.9: 3.552 s device vs 2.983 s host at latent length 20) by a different mechanism with the same cause: one accumulator per cell means it is f64-RATE bound rather than latency-bound, and Thor's consumer Blackwell runs fp64 at a fraction of its fp32 rate. The f64 stays. §13.2 records it as what all four consumers' goldens were taken with and what makes the CUDA provider memcmp-identical; narrowing it to torch's f32 re-gates four shipped models and is left OWED. The chain is broken bit-identically instead, by holding one f64 accumulator per cell over a TILE of output positions with the (ic, k) sweep hoisted outside it, so every cell receives the identical sequence of IEEE-754 double additions of the identical products in the identical order. The spec is committed before the implementation, and the row's issue is #1334, linked from `.agents/issue-index.md`, from §18 and from the pull request. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]

… f64 add chain per output cell (#672, #1334) `vt::cpu::Conv1dKernel` computed each output cell with ONE f64 accumulator swept over (ic ascending, k ascending). For a MiniMax-Music3 residual unit that is in_per_group * kernel = 384 * 7 = 2688 strictly dependent f64 additions per output element, so the loop had no instruction-level parallelism and could not be vectorised at any width. The measured rate is what a dependent `fadd` chain predicts and nothing else does: 1.7-2.0 GMAC/s on one core, ~2.8-3.0 cycles per multiply-accumulate on a 5.0 GHz Zen 5 whose `fadd` latency is 3. The kernel now holds 32 f64 accumulators, one per output position, and hoists the (ic, k) sweep outside them. Fix any single output cell and read what it receives, in order: the bias, then (ic=0,k=0), (ic=0,k=1), ... - the identical sequence of IEEE-754 double additions of the identical double products, in the identical order. The additions are INTERLEAVED across independent cells rather than serialised into one. That is a scheduling change and not an arithmetic one, so memcmp equality with the pre-change host loop, with the CUDA provider, and with all four consumers' goldens survives BY CONSTRUCTION rather than within a tolerance. The zero-padding skip becomes a CLAMP on the tile, which is the same set of (t, ic, k) triples: for a fixed k the in-range positions form one contiguous run. The constant trip count is not cosmetic. GCC's -O2 vector cost model is `very-cheap` and takes only a loop whose trip count is a known multiple of the vector width, and Release is -O3 while scripts/dgx-bringup.sh is RelWithDebInfo. Measured on the same source: a runtime-bounded accumulate loop gives 1.1-1.8x at -O2 and 5.1-5.4x at -O3; the whole-tile constant-trip path gives 5.2-5.8x at -O2 and 5.3-6.5x at -O3. `vt::ConvTranspose1d` gets the same treatment on its tap loop, where it is worth 2.7-2.9x at -O2 and nothing at -O3 - which is why that op is ~6 % of the chain's wall and Conv1d is ~94 %. stride > 1 keeps the shipped gather. Music3's Conv1d calls are all stride 1; the strided caller is the depthwise alias-free downsample, whose chain is one tap deep. `tests/vt/test_ops_conv1d_general.cpp` gains six tile-geometry cases and the forward cancellation case it did not have - its Conv1d cancellation arm compared CPU against CUDA only, so on a CPU-only build NOTHING held the forward sweep's ORDER against the pre-op host loop. RED-BEFORE, measured rather than argued: reversing the input-channel sweep in the tiled kernel (M1, compiled clean, binary sha 8bfd1eb4 against the green 5f71c5a0) FAILS this row's gate at 9 cases / 375 assertions and PASSES the base gate at 8 / 347. GREEN-AFTER 9 / 375, 0 failed. The reachability mutation (M2) breaks ONLY the whole-tile constant-trip path and reds `test_ops_conv1d_general` 11/375 and `test_host_parallel` 20/877, so the four consumers' own entry point does reach the new path rather than routing around it. `tools/bench/music3_vocoder_conv_ab.cpp` drives the production `VocoderDecode` that `Music3DecodeChunks` calls inside the `vocoder.decode_window` bracket, and `scripts/music3-vocoder-conv-ab.sh` builds two trees that differ in the kernel and in nothing else, refusing to time anything when the two binaries hash the same. docs/USAGE.md documents both. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]

The spec section this row appends lands at the end of a heavily contended file, so main's version is taken whole and 18 is re-applied after it rather than three-way merged. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]

Second merge of the row. The spec section 18 lands at the end of a heavily contended file, so main's version of that file is taken whole and 18 is re-applied after it rather than three-way merged. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]

… Thor, and the threadpool is what caps it (#672, #1334) Two `rc` leases on `thor:gpu0`, two independent builds, four binaries, and no `ssh` and no file mutex anywhere -- which is the thing §13.10's VOID speed axis did not have. Job `da3a2f94` (`--max-runtime 150m`) built both arms inside the lease from two clones in local `/tmp`, Release (-O3), CPU-only, and refused to time anything until the two binaries hashed differently (`d90e3912` vs `41ba78d2`; the trees differ in `src/vt/cpu/cpu_conv1d_general.cpp` and `diff -rq` says nothing else). Correctness first, on the after arm: `test_ops_conv1d_general` 9/375, `test_host_parallel` 8/877, `test_vocoder1d` 10/58, `test_bigvgan` 6/65, all `SUCCESS!` and rc=0. Three `[SKIP]` lines are read rather than ignored -- this worker has no `nvcc`, so every CPU-vs-CUDA arm did not run and the device `memcmp` re-measurement is OWED. Arms alternated, 3 rounds, best-of-3, the DEFAULT (host) arm. Medians: 1.364x at 20 latent frames, 1.414x at 40, 1.411x at 86, 1.424x at 172, 1.439x at 344. The weakest pair is kept rather than dropped, and it is the size §15.9 priced the device arm at. Flat over a 17x span of work, which is a rate change and not a fixed overhead. The waveform fingerprint is ONE value per length across all six arm-rounds, so the arms are bit-identical at full scale. Job `5b98f95e` (`--max-runtime 45m`) then split that number in a fresh container with a NEW pair of binaries. Single-thread: 37.0508 -> 17.1751 s, **2.157x**. The 14-thread control in the same container reproduces 1.365x to three digits. Scaling 1 -> 14 threads is 6.76x before and 4.27x after, of a possible 14x -- so neither arm scales and the FASTER one scales worse. The kernel is worth 2.16x per core; the threadpool returns 1.37x of it. That names the next lever as the vocoder's parallel decomposition (`ForOutputRows` partitions output channels, so all 14 threads sweep the whole input tensor) rather than as a ceiling, and it settles the aarch64-vs-x86 question: per core the same source is 2.16x here against ~5x on AVX-512, which the 4x narrower f64 vector explains. Two corrections ride with it. §13.10 recorded thor's worker as having "no compiler and no toolchain at all" and named a worker image as the blocker for measuring under a lease; probed 2026-08-19 it carries gcc, g++, cmake, ninja, make, python3 and git, and only `nvcc` is missing, so for a CPU arm that blocker does not exist. And the 53.6 s / 33.2 %-of-wall that defined this row is the CUDA arm -- §15.2's profile ran `VLLM_CPP_VOCODER_DEVICE=cuda` -- while the default is `cpu`, so the two are recorded side by side and never multiplied. No checkpoint was read: the driver uses synthetic weights at the shipped geometry, so §18.8's staging assertion is NOT APPLICABLE rather than skipped, and the e2e pair on the real checkpoint is owed. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]

…-> 19 Two rows claimed `## 17` of `.agents/specs/minimax-music3.md` concurrently, which the append-at-the-tail shape of that file cannot detect until they meet. #1315's request-key contract LANDED there first, so it keeps the number, and this row's depth-decoder section renumbers. It becomes 19 rather than 18 because the open #1356 (`MUSIC3-VOCODER-CONV-SPEED`) already claims 18 on its branch; taking 18 here would have produced exactly the same collision one merge later. Resolved by taking main's section whole and re-applying this row's edit beside it, never by a three-way merge of the two. Main's own §17.4/§17.7 back-references at :3526, :3606 and :3670 point INSIDE its section and are untouched. The 20 references to this row's section in the header, the two model TUs, the test and the three not-yet-landed issue-index rows all move to §19; the index rows for #1315, #1336 and #1337 are main's and are untouched, as is every §17 in `multimodal-speed.md`, `kimi-linear.md` and `benchmark-record.md`, which are those specs' own section 17. Nothing else in the incoming range touches a MiniMax-Music3 file, so the row's measured evidence stands as taken. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code]

…t bf16, and the measurement moves the blocker off the dtype (#672, #1309) (#1330) Row `MUSIC3-DEPTH-DEVICE`, issues [#1309](#1309) and [#672](#672); spec [`.agents/specs/minimax-music3.md`](.agents/specs/minimax-music3.md) §19. MiniMax-Music3's 0.646 B RVQ depth decoder was **48.4 % of a run** on `thor:gpu0` -- 78.1 s of 161 s, the largest single term -- so a 0.646 B model cost **6.3x** the 8.6 B language model beside it on the same box, and the reason was which processor each ran on. §16.4 measured the host kernel at ~4 bytes of f32 weight traffic per multiply-accumulate: 2.28 GB per call, 8 calls a frame. This row moves that sweep to the device and halves it. It is §11.4's last owed device row, and §14.5 blocked it on a dtype rather than on the work. ## The dtype, which is what the row was blocked on The oracle settles it by declaring nothing. `MiniMaxMusic3RVQDepthDecoder` takes no `dtype` parameter and contains no `torch.float32` literal and no `.float()` call (`diffusers` @ `c6da9936`, `minimax_music3_rvq_depth_decoder.py:101-125`); its single cast is the *down*-cast `:51` `.to(query.dtype)`. The dtype is imposed by `load_components(dtype=...)`, and `tools/oracle/music3_oracle.py` resolves `rvq_depth_decoder: torch.bfloat16` under both policies. So: **bf16 storage, f32 accumulation**, which is exactly the contract `vt::MatmulBT` has carried all along. §14.5's objection -- that an f32 `vt::MatmulBT` would drop the bf16 rounding every gated number was taken with -- is answered by using the bf16 one. The narrowing is **lossless** rather than a rounding we tolerate: `AtRuntimeDtype` already rounds every AR-half tensor through bf16 into an f32 carrier, so `vt::F32ToBF16` on those values is the exact inverse of the widening the loader did. A finding falls out that no gate here can see, and it is recorded rather than fixed: the *host* arm keeps bf16-exact weights in `std::vector<float>`, so every golden, token gate and WAV hash passes while that path moves twice the oracle's bytes (§19.2a). ## No new kernel Five shared ops, each already carrying a CPU **and** a CUDA provider: `vt::RmsNorm`, `vt::MatmulBT`, `vt::AttentionCross`, `vt::SiluAndMul`, `vt::Add`. Two merges the seam asks for: `to_q | to_k | to_v` as one `[3H, H]`, and `gate_proj | up_proj` as one `[2I, H]` consumed through `layers::UnquantizedMlpGateUpMethod` rather than a hand-rolled equivalent. The attention is causal upstream and `vt::AttentionCross` is still the right op, by an identity rather than a shortcut: the call presents **one** query row against a cache of positions that are all at or before it, so a causal mask masks nothing. ## What is measured, and what is not claimed No parity claim and no speed number. The decisive gate is `test_minimax_music3_ar_real` run with the device arm, and it needs the 28.5 GB checkpoint; until it runs, `AGENTS.md` forbids quoting a Thor A/B. **The A/B is blocked on correctness, not on the box.** ## The four review findings, repaired A fresh review returned FAIL. The port's algebra was not one of the findings -- the reviewer independently confirmed it exact. ### The reachability leg this row cites #1131 for was open one level up Two selections exist. The inner one, the `append` lambda, was gated and red. The outer one, the engine's `if (queue_.device.type != kCPU) { ... }`, was not: deleting the whole block -- the only thing `--speech-device 1` reaches -- left `test_minimax_music3_ar` 35/35 and `test_minimax_music3_speech` 9/9 green. That is #1131's shape reproduced by the change that names #1131 as its reason for existing, and it is structural rather than careless: on a CPU-only runner that condition can never be true. The rule now lives in `Music3SelectDepthArm`, which runs on **both** sides of the condition and is therefore drivable from a CPU gate. The gate enters it with a CPU queue -- which must stage nothing, keep the host arm, and release nothing even at `release_host=true` -- and with a fabricated queue on each of five non-CPU device types, where the invariant is that the selection **engages or refuses by name**. The third outcome, a quiet fall-back to the host loop, is the defect. `kCUDA` is excluded from that list deliberately: on a CUDA build without a device the staging would fail inside the CUDA runtime, and a call designed to fail latches a sticky error that the next unrelated kernel reports as its own. **What this does not close is stated rather than implied.** Deleting the engine's two-line *call* still leaves both suites green, because the engine needs the 28.5 GB checkpoint and a real device. §19.7 carries that as owed, naming row `MUSIC3-DEPTH-DEVICE` and #1131, and §19.5 records the mutation with the binary hashes that prove it was not a stale-binary artefact. ### The tolerance was fitted to one seed Changing only the RNG seed of the reference weights -- same distribution, same geometry, no defect -- reds the shipped `mean <= 4.0` on **three of six** equally valid draws. Worse, `worst <= 512` could not discriminate at all: a *correct* arm reads worst 7340 at `0xDEADBEEF` while the gate/up half swap reads 6641 and the wrong attention scale 6865, so any `worst` bound loose enough to admit a correct implementation admits two structural defects. The case now measures six seeds and gates the **median**, which is exactly **1 bf16 ULP at every seed** for the correct arm against a tightest defect of 3 -- a constant, not a distribution, so a redraw cannot move it. The mean stays gated at 15 inside a `(9.904, 19.6)` window that §19.4b states rather than hides. `worst` is reported and carries only a non-finite canary. Six seeds also exposed a defect in the metric itself: a reference value of exactly zero has no ULP -- bf16's spacing there is the denormal floor -- so one seed read mean 8.8e32 with the arm **correct**. Zero references now go in their own bucket, measured absolutely, and the two counts are asserted to **sum**, so a defect cannot hide by growing the un-gated bucket. ### The tolerance was blind to a too-wide dtype, on a row whose thesis is a dtype #1131 offers two instruments -- "invocation count **or resident dtype**" -- and this row had taken only the first. Widening one activation buffer to `kF32` left every case green while the path moved twice the bytes: `AGENTS.md`'s "a token gate cannot detect a dtype that is too wide" landing on this row exactly, and §19.2a's own finding about the host arm arriving inside the arm that exists to fix it. `Music3DepthDeviceResidentDtypes()` now reports a bit per dtype over every buffer the forward makes resident, read back by the gate rather than restated there. ### A stale upstream anchor `normalization.py:600-606` was wrong by ~46 lines and landed on `GlobalResponseNorm`. At the pinned `diffusers` `c6da9936` the two roundings are `normalization.py::RMSNorm.forward:557-561`, in the class the decoder actually constructs (`minimax_music3_rvq_depth_decoder.py:78,80,122`). Corrected in the header and both spec sites, with the symbol named. The copy in the append-only `.agents/issue-index.md` stays. ## A correction that outlives this row While repairing the above, §19.4's own reason 3 turned out to be wrong. It said `vt::RmsNorm` "deliberately drops a rounding" and that "the shared op is **wider** than the reference". Verified directly at the parity pin `555967922`: - `csrc/cpu/layernorm.cpp` computes `fp32_out = fp32_x * fp32_s_variance * fp32_w` and narrows once at `scalar_vec_t out(fp32_out)`; - `csrc/libtorch_stable/layernorm_kernels.cu:93` computes `static_cast<scalar_t>(x * s_variance * w)`; - upstream landed the weight-dtype multiply as [vllm#42379](vllm-project/vllm#42379) and **reverted** it as [vllm#46070](vllm-project/vllm#46070), which is an ancestor of the pin. So `vt::RmsNorm` mirrors its reference exactly. The mistake was generalising a `diffusers` observation onto a **shared op**: `AGENTS.md` makes vLLM the only reference wherever it implements the behaviour, and it implements RMSNorm, so diffusers was never the mirror source for the op -- only for the depth decoder *module*, which is what the host arm mirrors. Both arms are correct against the reference each answers to, and this term of the band therefore **does not close**. #1322's surviving half is `vt::SiluAndMul`, where `F.silu(gate)` really does produce a bf16 tensor before the multiply. ## The mutation battery Every run on a clean tree, with the compiler exit status, `git diff --stat` and the **binary** `sha256` printed, and the source restored and verified afterwards. | mutation | result | |---|---| | gate/up half swap | RED 2 cases / 14 assertions | | `q\|k\|v` merge order swapped | RED 2 / 15 | | wrong attention scale | RED 2 / 13 | | dropped position embedding | RED 2 / 15 | | K/V cache row collision | RED 2 / 14 | | activation buffer bf16 -> f32 | RED 1 / 1 | | K/V cache buffer bf16 -> f32 | RED 1 / 1 | | `Music3SelectDepthArm` gutted | RED 1 / 6 | | engine's call to the selector deleted | **GREEN** -- the residual, owed above | Gate at the head of this branch: `test_minimax_music3_ar` 37 cases / 640 assertions SUCCESS, `test_minimax_music3_speech` 9 / 223 SUCCESS, `check-fusion-consistency.py` rc=0 over 16 glue TUs and 12 gated-MLP TUs. `scripts/agent-preflight.sh --staged` is green except `test_cpu_x86_llamacpp_floor`, which is the load-dependent #618 red -- this diff touches no `scripts/` file and the box was at loadavg 251. ## Two smaller things, and one filed The **K/V cache** now draws from the device pool through `DBuf` rather than `backend.Alloc`/`Free`. It is a local of `Music3DepthStage`, so it is built and destroyed once per *frame*, and 16 raw `cudaMalloc` plus 16 synchronizing `cudaFree` per frame is what the pool exists to avoid. **No speed number is quoted**: it is unmeasured, and the A/B is blocked on correctness. #1351 is **filed rather than fixed**: `check-fusion-consistency.py` Check 2 runs `_MERGED_GEMM_SEAM` over `path.read_text()`, so a *comment* mentioning `MlpGateUpMethod` exempts a TU from the check. Reproduced by calling the checker's own functions -- the hand-rolled variant with comments kept reads `uses_merged_gemm_seam = True`, and flips to `False` only once comments are stripped. A checker change needs its own spec, a red-before test and green-after evidence, and widening the regex is what `AGENTS.md` forbids. ## One record collision, resolved and worth knowing about Two rows claimed `## 17` of `.agents/specs/minimax-music3.md` at the same time. #1315's request-key contract landed there first, so it keeps the number and this row's section renumbers to **§19** -- not 18, because the open #1356 (`MUSIC3-VOCODER-CONV-SPEED`) already claims 18 on its branch. Resolved by taking main's section whole and re-applying this row's beside it, never by a three-way merge; main's own back-references and every `§17` in `multimodal-speed.md`, `kimi-linear.md` and `benchmark-record.md` are untouched, as are the index rows for #1315, #1336 and #1337. The append-at-the-tail shape of that spec cannot detect this collision until the two branches meet, which is worth recording: a section number is a shared key in a file that has no lock. ## What is still owed The projection, audio embeddings, audio heads, CFG mix and top-k draw stay on the host -- ~1.6 % of the stage today, and the stage's remainder once the forward moves. The CUDA kernels themselves are unreached by any gate here, and so is the engine's own call to `Music3SelectDepthArm`: both need `thor:gpu0` and the 28.5 GB checkpoint, and §19.6 records them with #1131 owning the switch leg. The DiT arm one block below still carries in full the untestable shape this row just removed from the depth arm, and #1131 still owns that too. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [claude-code] --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io>

Third merge of the row, and the first that had to keep FOUR of main's sections of `.agents/specs/minimax-music3.md` rather than two. Main gained section 17 in d0598a2 and section 19 in c9e1b28 after this branch's last merge, so the end-of-file append conflicted again; `.agents/benchmark-record.md` conflicted the same way, both branches having appended a new entry at its end. Both are resolved by taking MAIN'S VERSION WHOLE and re-applying this row's addition beside it, never by three-way merging the file. Two end-of-file relocations auto-merge into a duplicate, so git refusing here is the safe outcome and a clean automatic merge would not have been. Section 18 lands between 17 and 19, which restores numeric order and collides with neither: 18 cites 13.2, 13.6, 13.10, 15.2, 15.9 and 16.7, all present, and neither 17 nor 19 cites 18. Verified programmatically rather than by reading: every one of main's 21 sections, `## Now` included, is byte-for-byte identical in the merged file, compared by sha256 per section. `.agents/benchmark-record.md` keeps main's 24,523-line prefix byte-identical and appends this row's 193-line entry after it. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]

…se never held, and withdraw three claims the evidence does not carry (#672, #1334) A fresh review of this row returned FAIL on four findings. The kernel is not one of them: the review reproduced bit-identity independently over 106,080 shape x data combinations with zero mismatches, 19,132 of them proven order-sensitive. Every repair here is in a test or a record. ## The guarantee was gated on ONE of its two axes `Conv1dKernel` hoists a `(ic ascending, k ascending)` sweep out of 32 accumulators, and reversing EITHER loop reassociates every cell. The cancellation case this row added has teeth along `ic` only, by construction: it pairs input CHANNELS and its own teeth check reverses `ic`. Measured on this tree, with the kernel's `k` loop reversed and nothing else changed: | | result | |---|---| | M3, `k` reversed, against the gate as this row shipped it | `test_ops_conv1d_general` 9/375, `test_host_parallel` 8/877, `test_vocoder1d` 10/58, `test_bigvgan` 6/65, every one `SUCCESS!` rc 0 (binary `c4bb0e76`, baseline `760061c5`) | | M3 against the gate with this commit's case | `test_ops_conv1d_general` **FAILURE**, 10 cases / 379 assertions, **1 failed**, 1552 of 1576 cells wrong in exactly that case (binary `eb25d992`) | | restored | 10/379 `SUCCESS!`, binary back to `f84d3a87`; the other three suites never moved off `efcff589` / `b01c6f57` / `b3ef384b` | The hole is OLDER than this row — the pre-change kernel with `k` reversed is green too — so it is not a regression this row introduced. It is still this row's to close, because this row is what turns that order into a load-bearing guarantee. `SerialConv1d` grows a `reverse_k` beside `reverse_ic`, and a second cancellation case pairs +2^40 against -2^40 across `k` inside one input channel held constant along its length, so the taps read the same value whatever `dilation` does with their positions. Sweeping `k` ascending cancels them at once and keeps the O(1) remainder exactly; descending accumulates the remainder first and quantises it at 2^-12, five orders above the f32 store's ULP at that scale. The clamp is driven THROUGH the pair: at t=0..1 both big taps are out of range, at t=2 only the negative one is, and from t=3 both are. ## Three claims the evidence does not carry **The CUDA f64-RATE mechanism was an inference written as a finding.** No counter, no fp64:fp32 ratio on the box, no CUDA A/B. What is checked is the kernel's shape; what is measured is a ratio flat at 0.37-0.40x over a 150x span, which rules out a fixed overhead and little else. Section 18.2 now separates checked, measured and inferred, and 18.9 carries the inference as owed with the three instruments that would settle it. **The rate arithmetic was not self-consistent.** 2.0 GMAC/s at 5.0 GHz is 2.5 cycles per MAC, not 2.8-3.0, and a strictly dependent chain of latency-3 `fadd`s cannot exceed 1.67 GMAC/s at all, so the top of the band sat above its own stated ceiling. The rate is corrected to the measured 1.76-1.86 GMAC/s. The cycles-per-MAC conversion is WITHDRAWN rather than restated, because the x86 host is a KVM guest reporting a nominal 4291.948 MHz with no `cpufreq` interface and `perf_event_paranoid` at 4: neither its boost clock nor a cycle count is observable from inside it, so every such figure was a conversion through an assumed clock. What survives is the identity — a dependent chain of latency-`L` adds cannot beat `clock / L` — and the 2.16x that breaking the chain bought. **5.2-5.8x at `-O2` is a KERNEL figure and read like a window one.** The decode window also runs `vt::ConvTranspose1d`, the alias-free activations, the strided downsamples and the pool around them; the review built the real project at `-O2` and measured 2.56x / 2.67x on the window. The clause is added in 18.4, and `docs/BENCHMARKS.md` no longer calls the single-thread 2.157x a kernel number, because it is the window on one thread. ## The two smaller ones `docs/STATUS.md` moved while the spec's `## Now` did not. The row does not change lifecycle state, so by the trigger table neither was owed; they are written together now and say so. And the A/B driver is an `add_executable` where its sibling is deliberately an OBJECT library: that is argued in place rather than left to be found twice, because `scripts/music3-vocoder-conv-ab.sh` builds it, `sha256sum`s both arms, refuses to time equal binaries and then RUNS them — every step needs a linked binary, and it is in no `install(TARGETS ...)`. `.agents/issue-index.md` is append-only and its #1334 row still carries the superseded rate sentence; the correction lives in the spec, which is where a correction belongs. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]

Fourth merge, taken because `origin/main` moved from 18f9948 to 601b576 while this row's review repairs were being written. Two files conflicted and both are the same shape as before: two branches appending at the end of one file, plus one keyed row both sides edited. `.agents/benchmark-record.md` takes MAIN'S VERSION WHOLE — its 24,757-line file is preserved byte-for-byte, verified by `diff` rather than by reading — and this row's 219-line entry is appended after it, also byte-for-byte. No three-way merge of the file, because two end-of-file relocations auto-merge into a duplicate and a clean automatic merge would have been the worse outcome. `docs/STATUS.md` had main's LTX-2.5 row rewritten under us and main's MiniMax-Music3 row edited on both sides. Main's LTX-2.5 row is taken whole. The Music3 row is main's, with this row's scoped edit re-applied on top of it rather than this branch's older copy restored: it keeps main's depth 4.45x and wall 2.74x and adds the vocoder decode window, inside the 220-character cell budget `scripts/check-public-doc-tables.py` holds and without raising a ratchet. `.agents/specs/minimax-music3.md` merged clean this time. Verified anyway, per section and by sha256: all 20 numbered sections of main, sections 15, 16, 17 and 19 included, are byte-for-byte identical here, and `## Now` differs only by this row's own added paragraph with zero lines of main's removed. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]

Fifth merge, and the only one taken for the FORGE rather than for the tree. `git merge-tree --write-tree HEAD origin/main` returns rc 0 and prints no conflict at 31f9378, while GitHub reports the pull request `CONFLICTING` / `DIRTY` on the same two commits. The difference is `.gitattributes`. `.agents/issue-index.md` carries `merge=union` so that two branches each appending a row merge instead of conflicting, and GitHub's merge does not apply the driver. Reproduced rather than assumed: `git merge-file` over the same three blobs with the DEFAULT driver exits 1 on that file alone. So the forge's verdict is an artifact of the driver and not a real textual conflict, and the fix is to materialise the union merge in a commit rather than to argue with the forge. Everything else merged clean. Verified per line rather than by reading: zero of `origin/main`'s issue-index lines are missing here, and the single line this branch adds that main does not have is its own #1334 row. The tree is unchanged by this merge apart from what main brought: no product source of this row is touched, so nothing here re-opens the gate the previous head reported. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]

Both sides appended to the append-only benchmark record. Resolved by keeping both, main's entry first and this row's after, so main's copy remains an ordered subsequence of the result. FOLLOWING_AGENTS_PROTOCOL Issue: #1334 Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]

mudler added 5 commits August 19, 2026 08:41

localai-bot mentioned this pull request Aug 19, 2026

feat(MUSIC3-DEPTH-DEVICE): the RVQ depth decoder reaches the device at bf16, and the measurement moves the blocker off the dtype (#672, #1309) #1330

Merged

mudler added 5 commits August 19, 2026 16:31

localai-bot merged commit c9724b5 into main Aug 19, 2026
1 of 3 checks passed

localai-bot mentioned this pull request Aug 20, 2026

MiniMax-Music3: the five changes main took on 2026-08-19/20 have NO end-to-end number — the depth device arm, the vocoder conv tiling and the conv f32 accumulator were each gated and none was timed on a real request (#672) #1512

Open

Navigation Menu

perf(MUSIC3-VOCODER-CONV-SPEED): the vocoder conv stops running one dependent f64 add chain per output cell (#672, #1334) - #1356

perf(MUSIC3-VOCODER-CONV-SPEED): the vocoder conv stops running one dependent f64 add chain per output cell (#672, #1334)#1356
localai-bot merged 10 commits into
mainfrom
row/MUSIC3-VOCODER-CONV-SPEED

localai-bot commented Aug 19, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

localai-bot commented Aug 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why it was slow

What changed, and why memcmp survives by construction

The gate, and the axis it did not hold

Speed — two rc leases, four binaries, no ssh and no file mutex

Stated plainly rather than implied

Gate

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

localai-bot commented Aug 19, 2026 •

edited

Loading

What changed, and why `memcmp` survives by construction

Speed — two `rc` leases, four binaries, no `ssh` and no file mutex