feat(MUSIC3-DEPTH-DEVICE): the RVQ depth decoder reaches the device at bf16, and the measurement moves the blocker off the dtype (#672, #1309)

[localai-bot]

Row MUSIC3-DEPTH-DEVICE, issues #1309
and #672; spec
.agents/specs/minimax-music3.md §19.

MiniMax-Music3's 0.646 B RVQ depth decoder was 48.4 % of a run on
thor:gpu0 -- 78.1 s of 161 s, the largest single term -- so a 0.646 B model
cost 6.3x the 8.6 B language model beside it on the same box, and the reason
was which processor each ran on. §16.4 measured the host kernel at ~4 bytes of
f32 weight traffic per multiply-accumulate: 2.28 GB per call, 8 calls a frame.
This row moves that sweep to the device and halves it. It is §11.4's last owed
device row, and §14.5 blocked it on a dtype rather than on the work.

The dtype, which is what the row was blocked on

The oracle settles it by declaring nothing. MiniMaxMusic3RVQDepthDecoder takes
no dtype parameter and contains no torch.float32 literal and no .float()
call (diffusers @ c6da9936,
minimax_music3_rvq_depth_decoder.py:101-125); its single cast is the down-cast
:51 .to(query.dtype). The dtype is imposed by load_components(dtype=...),
and tools/oracle/music3_oracle.py resolves rvq_depth_decoder: torch.bfloat16
under both policies. So: bf16 storage, f32 accumulation, which is exactly the
contract vt::MatmulBT has carried all along. §14.5's objection -- that an f32
vt::MatmulBT would drop the bf16 rounding every gated number was taken with --
is answered by using the bf16 one.

The narrowing is lossless rather than a rounding we tolerate: AtRuntimeDtype
already rounds every AR-half tensor through bf16 into an f32 carrier, so
vt::F32ToBF16 on those values is the exact inverse of the widening the loader
did. A finding falls out that no gate here can see, and it is recorded rather
than fixed: the host arm keeps bf16-exact weights in std::vector<float>, so
every golden, token gate and WAV hash passes while that path moves twice the
oracle's bytes (§19.2a).

No new kernel

Five shared ops, each already carrying a CPU and a CUDA provider:
vt::RmsNorm, vt::MatmulBT, vt::AttentionCross, vt::SiluAndMul, vt::Add.
Two merges the seam asks for: to_q | to_k | to_v as one [3H, H], and
gate_proj | up_proj as one [2I, H] consumed through
layers::UnquantizedMlpGateUpMethod rather than a hand-rolled equivalent.

The attention is causal upstream and vt::AttentionCross is still the right op,
by an identity rather than a shortcut: the call presents one query row against
a cache of positions that are all at or before it, so a causal mask masks nothing.

What is measured, and what is not claimed

No parity claim and no speed number. The decisive gate is
test_minimax_music3_ar_real run with the device arm, and it needs the 28.5 GB
checkpoint; until it runs, AGENTS.md forbids quoting a Thor A/B. The A/B is
blocked on correctness, not on the box.

The four review findings, repaired

A fresh review returned FAIL. The port's algebra was not one of the findings --
the reviewer independently confirmed it exact.

The reachability leg this row cites #1131 for was open one level up

Two selections exist. The inner one, the append lambda, was gated and red. The
outer one, the engine's if (queue_.device.type != kCPU) { ... }, was not:
deleting the whole block -- the only thing --speech-device 1 reaches -- left
test_minimax_music3_ar 35/35 and test_minimax_music3_speech 9/9 green. That is
#1131's shape reproduced by the change that names #1131 as its reason for
existing, and it is structural rather than careless: on a CPU-only runner that
condition can never be true.

The rule now lives in Music3SelectDepthArm, which runs on both sides of the
condition and is therefore drivable from a CPU gate. The gate enters it with a CPU
queue -- which must stage nothing, keep the host arm, and release nothing even at
release_host=true -- and with a fabricated queue on each of five non-CPU device
types, where the invariant is that the selection engages or refuses by name.
The third outcome, a quiet fall-back to the host loop, is the defect. kCUDA is
excluded from that list deliberately: on a CUDA build without a device the staging
would fail inside the CUDA runtime, and a call designed to fail latches a sticky
error that the next unrelated kernel reports as its own.

What this does not close is stated rather than implied. Deleting the engine's
two-line call still leaves both suites green, because the engine needs the
28.5 GB checkpoint and a real device. §19.7 carries that as owed, naming row
MUSIC3-DEPTH-DEVICE and #1131, and §19.5 records the mutation with the binary
hashes that prove it was not a stale-binary artefact.

The tolerance was fitted to one seed

Changing only the RNG seed of the reference weights -- same distribution, same
geometry, no defect -- reds the shipped mean <= 4.0 on three of six equally
valid draws. Worse, worst <= 512 could not discriminate at all: a correct arm
reads worst 7340 at 0xDEADBEEF while the gate/up half swap reads 6641 and the
wrong attention scale 6865, so any worst bound loose enough to admit a correct
implementation admits two structural defects.

The case now measures six seeds and gates the median, which is exactly
1 bf16 ULP at every seed for the correct arm against a tightest defect of 3 --
a constant, not a distribution, so a redraw cannot move it. The mean stays gated
at 15 inside a (9.904, 19.6) window that §19.4b states rather than hides.
worst is reported and carries only a non-finite canary.

Six seeds also exposed a defect in the metric itself: a reference value of exactly
zero has no ULP -- bf16's spacing there is the denormal floor -- so one seed read
mean 8.8e32 with the arm correct. Zero references now go in their own bucket,
measured absolutely, and the two counts are asserted to sum, so a defect
cannot hide by growing the un-gated bucket.

The tolerance was blind to a too-wide dtype, on a row whose thesis is a dtype

#1131 offers two instruments -- "invocation count or resident dtype" -- and
this row had taken only the first. Widening one activation buffer to kF32 left
every case green while the path moved twice the bytes: AGENTS.md's "a token gate
cannot detect a dtype that is too wide" landing on this row exactly, and §19.2a's
own finding about the host arm arriving inside the arm that exists to fix it.
Music3DepthDeviceResidentDtypes() now reports a bit per dtype over every buffer
the forward makes resident, read back by the gate rather than restated there.

A stale upstream anchor

normalization.py:600-606 was wrong by ~46 lines and landed on
GlobalResponseNorm. At the pinned diffusers c6da9936 the two roundings are
normalization.py::RMSNorm.forward:557-561, in the class the decoder actually
constructs (minimax_music3_rvq_depth_decoder.py:78,80,122). Corrected in the
header and both spec sites, with the symbol named. The copy in the append-only
.agents/issue-index.md stays.

A correction that outlives this row

While repairing the above, §19.4's own reason 3 turned out to be wrong. It said
vt::RmsNorm "deliberately drops a rounding" and that "the shared op is wider
than the reference". Verified directly at the parity pin 555967922:

• csrc/cpu/layernorm.cpp computes fp32_out = fp32_x * fp32_s_variance * fp32_w
  and narrows once at scalar_vec_t out(fp32_out);
• csrc/libtorch_stable/layernorm_kernels.cu:93 computes
  static_cast<scalar_t>(x * s_variance * w);
• upstream landed the weight-dtype multiply as vllm#42379 and reverted it as
  vllm#46070, which is an ancestor of the pin.

So vt::RmsNorm mirrors its reference exactly. The mistake was generalising a
diffusers observation onto a shared op: AGENTS.md makes vLLM the only
reference wherever it implements the behaviour, and it implements RMSNorm, so
diffusers was never the mirror source for the op -- only for the depth decoder
module, which is what the host arm mirrors. Both arms are correct against the
reference each answers to, and this term of the band therefore does not close.
#1322's surviving half is vt::SiluAndMul, where F.silu(gate) really does
produce a bf16 tensor before the multiply.

The mutation battery

Every run on a clean tree, with the compiler exit status, git diff --stat and
the binary sha256 printed, and the source restored and verified afterwards.

mutation	result
gate/up half swap	RED 2 cases / 14 assertions
q|k|v merge order swapped	RED 2 / 15
wrong attention scale	RED 2 / 13
dropped position embedding	RED 2 / 15
K/V cache row collision	RED 2 / 14
activation buffer bf16 -> f32	RED 1 / 1
K/V cache buffer bf16 -> f32	RED 1 / 1
Music3SelectDepthArm gutted	RED 1 / 6
engine's call to the selector deleted	GREEN -- the residual, owed above

Gate at the head of this branch: test_minimax_music3_ar 37 cases / 640
assertions SUCCESS, test_minimax_music3_speech 9 / 223 SUCCESS,
check-fusion-consistency.py rc=0 over 16 glue TUs and 12 gated-MLP TUs.
scripts/agent-preflight.sh --staged is green except
test_cpu_x86_llamacpp_floor, which is the load-dependent #618 red -- this diff
touches no scripts/ file and the box was at loadavg 251.

Two smaller things, and one filed

The K/V cache now draws from the device pool through DBuf rather than
backend.Alloc/Free. It is a local of Music3DepthStage, so it is built and
destroyed once per frame, and 16 raw cudaMalloc plus 16 synchronizing
cudaFree per frame is what the pool exists to avoid. No speed number is
quoted: it is unmeasured, and the A/B is blocked on correctness.

#1351 is filed rather than fixed: check-fusion-consistency.py Check 2 runs
_MERGED_GEMM_SEAM over path.read_text(), so a comment mentioning
MlpGateUpMethod exempts a TU from the check. Reproduced by calling the checker's
own functions -- the hand-rolled variant with comments kept reads
uses_merged_gemm_seam = True, and flips to False only once comments are
stripped. A checker change needs its own spec, a red-before test and green-after
evidence, and widening the regex is what AGENTS.md forbids.

One record collision, resolved and worth knowing about

Two rows claimed ## 17 of .agents/specs/minimax-music3.md at the same time.
#1315's request-key contract landed there first, so it keeps the number and this
row's section renumbers to §19 -- not 18, because the open #1356
(MUSIC3-VOCODER-CONV-SPEED) already claims 18 on its branch. Resolved by taking
main's section whole and re-applying this row's beside it, never by a three-way
merge; main's own back-references and every §17 in multimodal-speed.md,
kimi-linear.md and benchmark-record.md are untouched, as are the index rows
for #1315, #1336 and #1337.

The append-at-the-tail shape of that spec cannot detect this collision until the
two branches meet, which is worth recording: a section number is a shared key in
a file that has no lock.

What is still owed

The projection, audio embeddings, audio heads, CFG mix and top-k draw stay on the
host -- ~1.6 % of the stage today, and the stage's remainder once the forward
moves. The CUDA kernels themselves are unreached by any gate here, and so is the
engine's own call to Music3SelectDepthArm: both need thor:gpu0 and the 28.5 GB
checkpoint, and §19.6 records them with #1131 owning the switch leg. The DiT arm
one block below still carries in full the untestable shape this row just removed
from the depth arm, and #1131 still owns that too.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5-1m [claude-code]