diff --git a/.agents/issue-index.md b/.agents/issue-index.md index ce0034bf6..a585ac0be 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -427,6 +427,9 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#1325](https://github.com/mudler/vllm.cpp/issues/1325) | `ENG-RECORD-ANCHOR-RATCHET` | `scripts/record-anchor-baseline.json` stores a `"total"` that no code reads. `load_record_anchor_baseline` in `scripts/check-agent-record.py` returns `{bucket: int(data["buckets"][bucket]) for bucket in RECORD_ANCHOR_BUCKETS}` with `RECORD_ANCHOR_BUCKETS = ("stale", "broken")`, and it is the file's only reader; `check_record_anchors` iterates those two buckets, and `write_record_anchor_baseline`'s refusal compares `result.total > sum(previous.values())` — the buckets, not the stored `total`. Measured at `af87251c5`: mutating `"total": 38` to `39` (a file whose total disagrees with `32 + 6`) leaves `check-agent-record.py` exit 0, tree restored byte-for-byte by sha256. So this row's own budget file carries the exact shape the row exists to name: a recorded figure no gate reads, sitting beside the figures that are read and presenting as if it were checked. `--write-baseline` compounds it by printing `-> 38`, which reads as the value it stored and is the one no later run consults. TWO candidate resolutions, deliberately not chosen here because choosing belongs to the fixing row: read it and assert `total == stale + broken` on load, or drop the field and derive it at read time, which is the shape AGENTS.md §Records prefers. Either is a semantic change to `check-agent-record.py` owing a spec and a red-before case in `tests/scripts/test_agent_record.py` `RecordAnchorRatchet`. Distinct from [#1287](https://github.com/mudler/vllm.cpp/issues/1287) and [#1270](https://github.com/mudler/vllm.cpp/issues/1270), neither of which reaches the unread field. Also under `## Owed` in [`record-anchor-ratchet.md`](specs/record-anchor-ratchet.md) | bug | | [#1316](https://github.com/mudler/vllm.cpp/issues/1316) | — | `scripts/main-baseline.py` renders a scheduled run that executed ZERO jobs as `RED` with all 11 covered jobs `missing`, so `NEWEST BASELINE: RED at ` names a tree the run never checked out. Measured at `origin/main` `250db75a2`: runs `32206456661` and `32140419182` both return `startedAt: null` for every job, because GitHub cancelled them while they were pending in the single `ci-schedule-refs/heads/main-mudler/vllm.cpp` group, whose queue holds one run ([#274](https://github.com/mudler/vllm.cpp/issues/274)). Fail-closed, and the `missing (expected, never ran)` line is accurate about the jobs; the defect is the verdict word, because a run that executed nothing is NOT RUN rather than RED, and the newest verdict should fall through to the newest run that actually ran. NOT fixed in flow: `test_an_expected_job_the_payload_never_mentions_is_red` and `test_a_narrowed_run_reports_red_and_names_what_never_ran` deliberately assert missing-is-red so a narrowed run cannot pass, and separating "narrowed" from "never started" changes what the verdict means, which owes its own spec, red-before evidence and a fresh reviewer. Owed under `## Owed` of [baseline-lane-eviction.md](specs/baseline-lane-eviction.md), which removes the only observed producer of a zero-job run | bug | | [#1314](https://github.com/mudler/vllm.cpp/issues/1314) | `SPEC-DFLASH2` | **DFlash2 (`DFlash2DraftModel`) is unported**, and one config rule would run the published checkpoint wrong in silence. Upstream carries DFlash2 as a SECOND architecture beside DFlash rather than as a change to it ([vllm#52816](https://github.com/vllm-project/vllm/pull/52816), OPEN at head `19c9351904df4c63042671bc67a866ca48dc7d6f`, base `9842d701`, 755+/5-, 11 files, plus the stacked guard fix [vllm#52883](https://github.com/vllm-project/vllm/pull/52883)): DFlash1 gains two subclass seams and keeps every behaviour, and the new architecture adds a GROUPED DYNAMIC DEPTHWISE CONVOLUTION around each attention and each MLP sublayer plus a CANDIDATE SELECTOR that replaces the independent per-slot argmax with a scored path walk over the target head's top-K. Shapes taken from the published checkpoint rather than from the diff: `z-lab/Qwen3.8-27B-DFlash2`, safetensors header range-read 2026-08-19, 81 tensors -- DFlash1's set plus `layers.N.{attention,mlp}_conv.{base_kernel (2,2,5120), kernel_projection.weight (1280,5120)}` x5 and `candidate_selector.{hidden_projection.weight (256,5120), predecessor_codebook, successor_codebook}` at `(248320,256)` bf16 each, ~254 MB resident the DFlash1 lane never allocates; `conv_kernel_size 2`, `conv_group_size 16`, `selector_rank 256`, `selector_top_k 16`, `block_size 8`. **The silent one:** that config declares all five layers `sliding_attention` AND `is_causal false`, while our resolution mirrors the OLD upstream rule (causal iff SWA, unless `dflash_config.causal`, `include/vllm/model_executor/models/qwen3_dflash.h:22-24`), so every layer would run CAUSAL -- plausible tokens, a token gate against our own output sees nothing, and only ACCEPTANCE moves, which the lossless verify hides. Upstream changes `_dflash_layer_causal` to read `is_causal` first, in the same commit. Three further things are owed and none of them is silent: no route for the `DFlash2DraftModel` architecture string (the same classification code as the open `DSparkDraftModel` gap, [#1193](https://github.com/mudler/vllm.cpp/issues/1193)); no top-k that EMITS the surviving (id, value) pairs, where the decision is to extend the sort-free pivot-bracket threshold search already ported from the same FlashInfer approach at `src/vt/cuda/cuda_sample.cu:297-506` rather than port FlashInfer's 3380-line general radix kernel; and the path walk must run ON DEVICE from the first landing, because the identical sequential shape in DSpark shipped host-side and measured 28% of the 27B draft step ([#436](https://github.com/mudler/vllm.cpp/issues/436)) before `SampleSequentialDevice` moved it. Already reusable unchanged: `vt::DFlashBlockAttention`, the DFlash runner/rejection/GDN-rollback lane, and the loader's target-shared `embed_tokens`+`lm_head`, which is already what a DFlash2 checkpoint needs. BEYOND-PIN by developer decision 2026-08-19 (mirror the open PR now, reconcile if review moves it), in the same posture `SPEC-DSPARK-QWEN3-ROUTING` takes toward vllm#52197; the parity pin `555967922` does not carry the architecture and is NOT advanced. Gate arm is bf16 27B plus the GGUF drafter arm in the same wave, oracle = vLLM built at the PR head, acceptance measured SAME-TRAJECTORY because `SPEC-DFLASH` D8 spent a campaign on a divergent-trajectory confound that D9 refuted. Spec [dflash2-spec-decode.md](specs/dflash2-spec-decode.md) | feature | +| [#1309](https://github.com/mudler/vllm.cpp/issues/1309) | `MUSIC3-DEPTH-DEVICE` | MiniMax-Music3's 0.646 B RVQ depth decoder is **48.4 % of a run** on `thor:gpu0` and is the last large stage still on the host, so a 0.646 B model costs **6.3x** the 8.6 B language model beside it on the same box. §11.4's last owed device row: the vocoder closed in §13, the DiT in §14, and §14.5 blocked this one on a dtype rather than on the work. **The dtype is settled in [§19.2](specs/minimax-music3.md): bf16 storage, f32 accumulation.** The oracle settles it by declaring nothing — `MiniMaxMusic3RVQDepthDecoder` takes no `dtype` parameter and contains no `torch.float32` literal and no `.float()` call (`diffusers` @ `c6da9936`, `models/transformers/minimax_music3_rvq_depth_decoder.py:101-125`); its one cast is the **down**-cast `:51` `.to(query.dtype)`; and `tools/oracle/music3_oracle.py:91-95,103-108` resolves `rvq_depth_decoder: torch.bfloat16` under both policies. `vt::MatmulBT` already carries exactly that contract (`include/vt/ops.h:1281-1283`), so §14.5's objection — that an **f32** `vt::MatmulBT` would drop the bf16 rounding every gated number was taken with — is answered by using the **bf16** one. A finding falls out that no gate here can see: the host arm keeps bf16-exact weights in `std::vector`, so every golden, token gate and WAV hash passes while the path moves twice the oracle's bytes ([§19.2a](specs/minimax-music3.md)) | feature | +| [#1322](https://github.com/mudler/vllm.cpp/issues/1322) | `MUSIC3-DEPTH-DEVICE` | `vt` cannot express torch's **per-op bf16 rounding**, and no caller can build it. A bf16 torch module rounds at every op boundary; `vt::RmsNorm` keeps full f32 across the weight multiply (`include/vt/ops.h` says so) where `normalization.py:600-606` casts back to the weight dtype first, and `vt::SiluAndMul` computes `silu(g)*u` wholly in f32 where `F.silu(gate)` produces a **bf16 tensor** that is then multiplied. MEASURED on MiniMax-Music3's depth decoder ([#1309](https://github.com/mudler/vllm.cpp/issues/1309), spec §19.4a): the device arm reads worst **110 bf16 ULP** / mean **2.095** against the host reference, and collapsing those two host roundings to the seam's shape takes it to worst 8 / mean 0.0596 and the composed stage to **ZERO — bit-identical**. So ~97 % of the deviation is rounding POLARITY inside two shared ops, not dtype, accumulator or reduction order. A call site cannot split them, because `vt` has no elementwise or row-broadcast multiply — `kMulScalar` takes a scalar and `kMulColVecF32` is an f32 in-place column scale. Closing it needs either a `vt::Mul` with `vt::Add`'s two operand shapes or rounding-faithful modes on the two ops, with CPU and CUDA providers; #1309 records the measurement and stops. A token gate cannot see any of this | bug | +| [#1351](https://github.com/mudler/vllm.cpp/issues/1351) | `MUSIC3-DEPTH-DEVICE` | `scripts/check-fusion-consistency.py` Check 2 is satisfied by a **COMMENT**: it runs `_MERGED_GEMM_SEAM` over `path.read_text()` with no comment stripping, so a model TU that merely *mentions* `MlpGateUpMethod` in prose is exempt from the check forever. Measured on `minimax_music3_depth_device.cpp` ([#1309](https://github.com/mudler/vllm.cpp/issues/1309), spec §19.7): replacing the `layers::UnquantizedMlpGateUpMethod` call with an inline hand-rolled `vt::MatmulBT` + `vt::SiluAndMul` path while leaving the two comment mentions gives `uses_merged_gemm_seam = True` and the checker returns **rc=0 green**, with the numbers bit-identical so no other gate sees it; stripping comments flips it to `False`. The only comment-aware helper in the file, `allowlisted_names`, strips `#` comments from the allowlist FILE rather than from the scanned source. NOT a defect: `MlpGateUp[A-Za-z]*Method` matching a renamed `MlpGateUpXX` is deliberate, to cover `UnquantizedMlpGateUpGeluMethod`. Fixing it is a semantic checker change — spec, red-before test, green-after evidence — and must not be done by widening the regex; stripping comments also moves the site count, so a fixture must pin both numbers | bug | | [#1315](https://github.com/mudler/vllm.cpp/issues/1315) | `MODEL-MUSIC-minimax-music3-mini-max-music3-for-conditional-generation` | `POST /v1/audio/speech` dropped keys in TWO ways that [#925](https://github.com/mudler/vllm.cpp/issues/925) and [#953](https://github.com/mudler/vllm.cpp/issues/953) could not reach. **(a) `extra_params` REPLACED the top level rather than adding to it.** `speech_api.cpp:160-163` binds `extra` to ONE of the two objects, so a body carrying an `extra_params` object at all — an empty `{}` suffices — stopped the top level being read for every knob and every refusal resolving through it. `{"lyrics":…,"extra_params":{"seed":7},"audio_duration":0.1}` dropped the duration, left `audio_duration_s` at its `0.0` sentinel, and `minimax_music3_speech.cpp:478-479` substituted the family's 60 s: #852's ~750x job behind a 200, with #925's guard in the tree and unable to fire. The same body silenced all five #953 refusals, and the mirror hole dropped `voice`/`speed`/`stream`/`stream_format`/`response_format` whenever they were NESTED. The video route already resolves this correctly and documents the precedence (`video_api.cpp:216-225`). **(b) Nine more keys the pinned SGLang-Omni oracle `748a0b43` refuses BY NAME were dropped silently**: the seven of `_UNSUPPORTED_TTS_PARAMS` this route can see (`task_type`, `ref_audio`, `ref_text`, `x_vector_only_mode`, `initial_codec_chunk_frames`, `token_count`, `duration_tokens` — `request_builders.py:20-30,71-81`, forwarded by `speech_service.py:737-779`), plus `speaker` (upstream's declared ALIAS for `voice`, `protocol.py:337-339`, so one spelling was refused and the other returned 200) and `instructions` (upstream's spelling of the music CAPTION, required non-empty at `request_builders.py:104-106`, so a caller porting the upstream recipe lost the whole music description). `token_count`/`duration_tokens` are LENGTH keys and repeat #925's cost exactly. `language` is deliberately NOT in the set: it is already refused by name one layer down at `minimax_music3_speech.cpp:456-460`, where a family that HAS a language can still take it. FIXED IN FLOW: one `owner(key)` resolver for both placements, nine refusals naming what to send instead, #925's forward-compatibility boundary kept and pinned by a negative control; RED first through the registered route and over a real socket, reachability proven by deleting the parse call site. Spec [§17](specs/minimax-music3.md) | bug | | [#1336](https://github.com/mudler/vllm.cpp/issues/1336) | `MODEL-MUSIC-minimax-music3-mini-max-music3-for-conditional-generation` | `POST /v1/audio/speech`: [#1315](https://github.com/mudler/vllm.cpp/issues/1315) routed the KNOBS and the REFUSALS through the `Owner(key)` lookup that sees both placements, and left the eight CONTENT keys on the bare `json` handle (`speech_api.cpp:126,127,133,134,136,138,140,141,144,145,175,176` @ `7d21644d7`): `model`, `input`, `text`, `language`, `lyrics`, `description`, `prompt`, `reference_audio`. So nesting a content key under `extra_params` still DROPPED it, and that DEFEATED three refusals: `{"extra_params":{"text":"hello"}}` returned 200 instead of `minimax_music3_speech.cpp:440-446`, whose refusal ends with the words "rather than having it silently dropped"; `{"extra_params":{"language":"en"}}` returned 200 instead of `:456-460`; and `{"extra_params":{"reference_audio":"data:…"}}` dropped the clip, so a family whose `requires_reference_audio()` is true answered `400 "reference_audio … is required"` (`api_server.cpp:511-516`) to a caller who had supplied one. NOT a regression, because the pre-#1315 code read those keys from the top level only as well. What let it survive an implementation and a first review was the FALSE structural claim #1315 left at `speech_api.cpp:110-113` — "there is no longer a handle on one placement only", when `Owner` needs `json` as its fallback and cannot remove it. FIXED IN FLOW while repairing the review of [#1328](https://github.com/mudler/vllm.cpp/pull/1328): the content keys route through `Owner`, the comment claims a convention the tests pin rather than an impossibility, and nested `text`/`language`/`reference_audio` are RED first at the parser and at the registered route. #925's boundary is unchanged and a negative control in BOTH placements pins it. Spec [§17.7](specs/minimax-music3.md) | bug | | [#1337](https://github.com/mudler/vllm.cpp/issues/1337) | `MODEL-MM-indextts2-index-tts2-talker-for-conditional-generation` | IndexTTS-2 drops `language` behind a 200. `SpeechGenParams::language` is documented as "upstream's `lang`; empty => the family's default" (`include/vllm/multimodal/speech_engine.h:76`), and `grep -rn '\.language' src/vllm/model_executor/models/` returns exactly ONE hit: `minimax_music3_speech.cpp:456`, which reads it only to REFUSE it. `grep language src/vllm/model_executor/models/indextts2.cpp` returns nothing, so `{"input":"ciao","language":"it"}` against an IndexTTS-2.5 server parses, reaches the family, is dropped, and returns audio in whatever language the model inferred, with no way for the caller to learn the key was ignored: [#925](https://github.com/mudler/vllm.cpp/issues/925) exactly, in the family that row did not touch. The model is NOT indifferent to the field — `indextts2_talker_loader.cpp:164` reads `lang_embedding.weight` into `out.languages`, so the checkpoint carries an embedding the request path never selects with. This also corrects a justification: `.agents/specs/minimax-music3.md` §17.4 argued `language` stays refused one layer down because "moving it up would break IndexTTS-2.5", which is not true today (no family reads it, so moving it up would convert this silent drop into a refusal); §17.4 now carries the forward-looking form instead. Two acceptable fixes: wire `language` into the talker, or refuse it by name at the IndexTTS-2 family layer until that lands. NOT fixed in [#1328](https://github.com/mudler/vllm.cpp/pull/1328), which is scoped to `ParseSpeechRequest`; either fix is IndexTTS-2 model code with its own oracle reading against the vLLM-Omni TTS lane | bug | diff --git a/.agents/specs/minimax-music3.md b/.agents/specs/minimax-music3.md index 9deab5baa..dedc8f5cb 100644 --- a/.agents/specs/minimax-music3.md +++ b/.agents/specs/minimax-music3.md @@ -3679,3 +3679,527 @@ review did not find the drops by reading the diff. It found them by testing the INVARIANT the diff asserted, and the invariant was the only part of the change that had no test. A claim about what can no longer be written is a claim, and it gets mutated like any other. + +--- + +## 19. The depth decoder reaches the device (#672, [#1309](https://github.com/mudler/vllm.cpp/issues/1309)) — §11.4's LAST owed device row, and the dtype it was blocked on + +§11.4 named three device rows. The vocoder closed in §13, the DiT in §14, and +§14.5 left this one blocked on a dtype rather than on the work. §16 then removed +the arithmetic nobody read, which made this row **worth 4.4x less than when it +was written** and did not make it unnecessary: on `thor:gpu0` the depth forward +is still **48.4 % of a run**. + +This section settles the dtype and describes the arm. The row is +`MUSIC3-DEPTH-DEVICE`. + +### 19.1 The gap, measured, on the head this row starts from + +`main` at `678fc672c`, `thor:gpu0` sm_110, `--device 1 --duration 4 --steps 4`, +100 frames, checkpoint staged to local disk (so §15.6's cold-CIFS load is not +inside these figures): + +| bucket | seconds | share of wall | +|---|---|---| +| `ar.depth_forward` | 78.1 | **48.4 %** | +| `vocoder.decode_window` | 53.6 | 33.2 % | +| `ar.lm_decode_step` | 12.3 | 7.6 % | +| `denoise.dit_device` | 5.1 | 3.1 % | + +A **0.646 B** decoder costs **6.3x** the **8.6 B** language model beside it. The +sizes are not the reason; which processor each runs on is. §16.4 measured the +host kernel at ~4 bytes of f32 weight traffic per multiply-accumulate — 2.28 GB +per `DepthDecoderAppend` call, 8 calls a frame — and §16.6b measured 96.93 ms +per call, i.e. **~23.5 GB/s achieved** across Thor's 14 cores. + +### 19.2 THE DTYPE — settled, and the oracle settles it by declaring nothing + +§14.5 blocked this row because routing an `ArCompute::kBFloat16` decoder through +an **f32** `vt::MatmulBT` would silently drop the rounding every gated number was +taken with. That reasoning is correct and its conclusion has expired, because +the row does not need an f32 `vt::MatmulBT`. It needs a **bf16** one, and +`vt::MatmulBT` has been one all along: + +> `a/b bf16 (or f32), out f32 or bf16, contiguous, same device` — same f32 +> accumulation and dtype contract as `Matmul` +> ([`include/vt/ops.h:1281-1283`](../../include/vt/ops.h)) + +The CUDA provider states the same contract as a refusal by name +(`src/vt/cuda/cuda_matmul.cu:296-300`): `supported: (bf16,bf16)->f32|bf16, +(f32,f32)->f32|bf16`. **Mixed f32-activation × bf16-weight is rejected**, so the +arm is bf16 on both operands or it is not bf16 at all. + +**What the oracle says.** `MiniMaxMusic3RVQDepthDecoder` declares **no dtype**. +Its `@register_to_config __init__` takes no `dtype` parameter and constructs every +submodule at torch's ambient default +(`diffusers` @ `c6da9936`, +`models/transformers/minimax_music3_rvq_depth_decoder.py:101-125`). There is **no +`torch.float32` literal and no `.float()` call anywhere in the file**. The single +cast in the module is a **down**-cast that re-aligns the attention output to the +query dtype: + +```python +51 hidden_states = hidden_states.flatten(2, 3).to(query.dtype) +``` + +So the dtype is imposed from outside, by `load_components(dtype=...)`, and +[`tools/oracle/music3_oracle.py:91-95,103-108`](../../tools/oracle/music3_oracle.py) +resolves `rvq_depth_decoder: torch.bfloat16` under **both** its policies — +`ON_DISK_DTYPES` and `REFERENCE_DTYPES` differ only in `condition_encoder`. That +is the invariant §2.1 already records at :176 and :179-182: +`dtype(language_model) == dtype(rvq_depth_decoder) == dtype(condition_encoder)`, +bf16 AR half over fp32 acoustic half. + +**Where fp32 legitimately appears, and why it is not this row's.** The depth loop +promotes to fp32 exactly twice, both **after** the decoder and both on logits: +the CFG mix at `encoders.py:134` (`logits[:1].float()`) and the sampler at +`encoders.py:95` (`torch.nan_to_num(logits.float(), ...)`). Both already run on +the host in `Music3DepthStage`, and this row does not move them. + +**The decision.** Weights bf16, activations bf16, every op boundary storing bf16, +accumulation f32. That is `vt::MatmulBT`'s contract unmodified, and it is what a +bf16 torch module on a GPU does. + +#### 17.2a The finding this makes visible, which no gate in this tree can see + +The host arm holds the depth decoder's weights in `std::vector`. The +checkpoint is bf16 and the loader widens it, so every stored value is +bf16-exact: the numbers are right, and the goldens, the token gates and §16.6b's +WAV hash all pass **while the path moves twice the bytes the oracle moves**. + +That is `AGENTS.md`'s "a token gate cannot detect a dtype that is too wide", in +this tree, on the stage that is half the run. §16.7 records the same fact from +the other side as lever 1. It is written here as a **finding**, not as a defect +introduced by this row: the host arm's f32 containers are what W2/W3's +reduction-order gates were taken against, and this row does not narrow them. The +device arm is the narrow one. + +Per [`.agents/porting.md`](../porting.md) "Mirror the memory format, not just the +math", the four questions and their answers for this path: + +| ask | answer, and where upstream answers it | +|---|---| +| what dtype does the linear OUTPUT? | bf16 — `nn.Linear` at module dtype, no cast (`minimax_music3_rvq_depth_decoder.py:63-66,114,124`) | +| what dtype do the intermediate activation buffers carry? | bf16 — read the consumer: `:51` casts the attention output back **to** `query.dtype` | +| is a projection one physical GEMM or several? | several upstream (`to_q`/`to_k`/`to_v`, `gate_proj`/`up_proj`); **merged here**, see §19.3 | +| what `kv_cache_dtype` is resolved? | not applicable — the depth cache is this decoder's own 16-position table, not a paged KV cache; it is bf16 with its activations | + +### 19.3 The design — no new kernel, and two merges the seam asks for + +Every op already exists with a **CPU and a CUDA provider**, so this row composes +a forward rather than adding a kernel. That is §14.2's shape and it is deliberate. + +| host reference (`minimax_music3_ar.cpp`) | shared op | +|---|---| +| `RmsNorm` | `vt::RmsNorm` | +| `LinearNoBias` | `vt::MatmulBT` | +| `CausalAttentionStep` | `vt::AttentionCross`, `bias = nullptr` | +| `silu(gate) * up` | `vt::SiluAndMul` | +| residual adds | `vt::Add` | + +**The attention is causal upstream and `vt::AttentionCross` is the right op +anyway, for a reason that is an identity rather than a shortcut.** +`DepthDecoderAppend` presents **one** query row against a cache of `seq` +positions that are all at or before it. A causal mask over a single query at the +last position masks nothing. So the non-causal op computes upstream's causal +result exactly, and `vt::Attention` — which would apply a mask keyed on a query +index this call does not have — is the wrong one. Upstream's own +`dispatch_attention_fn(..., is_causal=True)` (`:43-50`) runs whole sequences; +ours runs one row, and that is §16.3's identity, already gated bitwise. + +**Two merges, because AGENTS.md routes mergeable projections through the merged +seam and because they cost nothing numerically.** Each output element remains its +own dot product, so merging changes no reduction: + +* `to_q | to_k | to_v` stage as one `[3H, H]` weight; one `vt::MatmulBT` produces + `[batch, 3H]`, split by re-view (`vt::QkvSplit` is available but a free + `MakeTensor` at an offset is what the DiT arm uses and needs no copy). +* `gate_proj | up_proj` stage as one `[2I, H]` weight. This is the merge that + `vt::SiluAndMul` **requires**: it computes `silu(x[:, :D]) * x[:, D:]` over one + `[T, 2D]` buffer, and upstream's `silu(gate_proj(x)) * up_proj(x)` puts gate + first. **No half swap** — unlike §14.3's DiT, where the assignment was the + opposite way round and needed a stage-time exchange. + +The weight sweeps per frame fall from 8 (decoder) + 7 (projection) of f32 to the +same counts at **half the bytes**, and the sweep itself moves from 14 host cores +to the device. + +**What does NOT move.** The audio heads, the CFG mix, the top-k draw, the +feedback embedding and the projection of the fed-back row stay on the host in +this row, exactly as `Music3DepthStage` has them. The heads are seven +`[1024, 4096]` GEMVs a frame against the decoder's 8 sweeps of 570 M +multiply-accumulates; §15's profile puts `ar.depth_projection` at 1.307 s and +the forward at 78.316 s. Moving them is owed, not skipped silently: §19.7. + +### 19.4 Correctness — bitwise is NOT achievable, said before the code, with the tolerance and its reason + +**This arm cannot be bit-identical to the host arm, and no gate here will claim +it.** §16 could claim it because a causal identity is exact. This row changes the +machine. Three independent reasons, each sufficient on its own: + +1. **The accumulator narrows.** The host keeps a sequential `double` per output + element (`minimax_music3_ar.cpp::LinearNoBias`); `vt::MatmulBT` accumulates in + **f32** (`CUBLAS_COMPUTE_32F`). Over `in_dim` 4096 and 6144 that is a real + difference, not a formality. +2. **The reduction re-associates.** cuBLASLt splits K by an algorithm this row + does not choose, and `vt::AttentionCross`'s CUDA kernel uses an **online + softmax recurrence** where the host and the CPU provider use a three-pass + max/sum/weighted-sum. The two are not bit-identical to each other either. +3. **The two arms normalize against different references, and both are right.** + The Music3 **host** arm's own RmsNorm (`minimax_music3_ar.cpp`) mirrors the + diffusers module this model *is*: `normalization.py::RMSNorm.forward` casts + back to the weight dtype at `:560` and then multiplies at `:561`, so it rounds + **twice**, and the decoder constructs exactly that class — `class RMSNorm` at + `:510`, built at `minimax_music3_rvq_depth_decoder.py:78,80,122`. (`:600-606` + is `GlobalResponseNorm` and `:590-597` is `MochiRMSNorm`; neither is on this + path. The earlier `:600-606` citation in this spec was wrong by ~46 lines.) + The **device** arm routes through the shared `vt::RmsNorm`, which mirrors + vLLM, and vLLM keeps f32 across the weight multiply and rounds **once**. + + **`vt::RmsNorm` does not diverge from its reference, and an earlier revision of + this section said it did.** Verified directly at the parity pin `555967922`: + `csrc/cpu/layernorm.cpp` computes `fp32_out = fp32_x * fp32_s_variance * + fp32_w` and narrows once at `scalar_vec_t out(fp32_out)`; + `csrc/libtorch_stable/layernorm_kernels.cu:93` computes + `static_cast(x * s_variance * w)`. Upstream landed the + weight-dtype multiply as vllm#42379 and **reverted** it as vllm#46070, which is + an ancestor of the pin. `AGENTS.md` makes vLLM the only reference wherever it + implements the behaviour and it implements RMSNorm, so diffusers was never the + mirror source for the shared op — only for the model. + + What follows is that this term of the band will **not** go away, because + neither side is defective. Which rounding is right *for this model* is a + question the diffusers oracle settles through `test_minimax_music3_ar_real`, + and §19.6 records that as owed. + +**The gate is therefore a tolerance, and the tolerance is derived rather than +chosen.** The proposal, to be replaced by the measured value before the row is +`DONE`: + +* the reference is the **host arm at `ArCompute::kBFloat16`**, not an f32 forward, + because the host arm is what every committed number was taken with; +* the bound is stated in **bf16 ULPs of the reference value**, not in absolute + units, because the activations span orders of magnitude across four layers; +* the proposed acceptance is **max 2 bf16 ULP** with a **mean below 0.5 ULP** + over every compared value, which is the band §5's own matched control already + established for this decoder (the header records ~1.3 bf16 ULP average between + an f32 and a bf16 forward of the same weights); +* the gate asserts its own **teeth**: that the compared reference values are + non-zero and that a deliberately wrong arm exceeds the bound. A tolerance + nothing can fail is not a gate. + +**A tolerance gate cannot see a dropped stage, so it is not the only gate.** The +composition is held by the same instrument §16.5 had to add: the production +`Music3DepthStage` is driven end to end and its **drawn codes and draw count** +are compared, which a numeric tolerance would not notice. + +### 19.4a The tolerance was MEASURED, and it refuted §19.4's own ordering + +§19.4 named three sources and led with the accumulator. **The measurement puts +that last and it is corrected here rather than quietly left standing.** + +`test_minimax_music3_ar`, 8 heads of 8, three layers, the full 8-position +schedule, bf16-exact pseudo-random weights, on a **CPU `vt::Queue`**: + +| arm | worst | mean | composed stage, worst | +|---|---|---|---| +| as shipped | **110 bf16 ULP** | **2.095** | **255 bf16 ULP** | +| host's two intermediate roundings COLLAPSED to the seam's shape | 8 ULP | 0.0596 | **0 — bit-identical** | + +The second row is an attribution mutation, applied to the **host** reference and +restored `sha256`-verified. It removes exactly two roundings: + +1. The **host** arm's cast back to the weight dtype **before** the affine + multiply — `Store(Store(normed) * w)` becomes `Store(normed * w)`. That is + `normalization.py::RMSNorm.forward:559-561` — the `.to(self.weight.dtype)` at + `:560` and the `* self.weight` at `:561` — which the host arm mirrors on + purpose because it mirrors the diffusers module, and which `vt::RmsNorm` does + not do because it mirrors vLLM, where the multiply is f32 and the narrowing is + single (§19.4 reason 3, verified at the pin). **Neither side is defective**, + so this term does not close. +2. SiLU's own store before the `* up` multiply — `Store(Store(silu) * up)` + becomes `Store(silu * up)`. torch computes `F.silu(gate)` into a **bf16 + tensor** and then multiplies, so upstream rounds there too; `vt::SiluAndMul` + computes the whole expression in f32 and rounds once. + +**So ~97 % of the mean deviation is rounding POLARITY inside two shared ops, not +the accumulator, not the reduction order and not the dtype.** The accumulator and +reduction-order terms §19.4 led with are the 0.0596 ULP remainder, which is the +order of magnitude first principles predict for f32-vs-f64 accumulation over 64 +terms. §19.4's reason (3) was right and was ranked third; reasons (1) and (2) are +real and are nearly invisible beside it. + +**The counterfactual is the strongest statement this row has: with the roundings +aligned, the composed device stage is BIT-IDENTICAL to the host arm over 448 +values.** The port's algebra — the merged `[3H, H]` and `[2I, H]` layouts, the +cache indexing, the batch-2 sequencing, the attention identity — is therefore +exactly right, and the entire remaining difference is rounding polarity in two +shared ops — one of which (`vt::SiluAndMul`) is a genuine seam gap and one of +which (`vt::RmsNorm`) is two references legitimately disagreeing. See §19.4 +reason 3, which corrects an earlier claim in this spec that both were seam gaps. + +#### What this changes about the row, said plainly + +**The blocker was never the dtype, and it is not the dtype now.** §14.5 named the +dtype; §19.2 settled it and the settlement stands. What the measurement exposes +is a different obstacle, and **it is half the size this section first claimed**. + +**The `vt::RmsNorm` half is not a seam gap at all.** §19.4 reason 3 carries the +verification: vLLM's own RMSNorm multiplies in f32 and narrows once, on both the +CPU and the CUDA path at the parity pin, and upstream reverted the weight-dtype +variant. `vt::RmsNorm` therefore mirrors its reference exactly. The two arms +differ because the *host* arm mirrors the diffusers module and the *device* arm +mirrors vLLM, and both are correct against the reference each answers to. That +term stays in the band permanently, and no seam extension removes it. + +**The `vt::SiluAndMul` half is real.** `F.silu(gate)` produces a **bf16 tensor** +that is then multiplied, where `vt::SiluAndMul` computes the whole expression in +f32 and rounds once. It cannot be worked around at the call site, because `vt` +has no elementwise or row-broadcast multiply — `MulScalar` takes a scalar and +`MulColVecF32` is an f32 in-place column scale — so expressing +`Store(silu) * up` needs a seam extension with CPU and CUDA providers and its own +gates. That is [#1322](https://github.com/mudler/vllm.cpp/issues/1322)'s +surviving half, it is its own row, and it is NOT taken here. + +**What is consequently NOT claimed.** No parity claim, and no speed number. The +row's numeric gate bounds an attributed divergence; it does not prove agreement +with the reference. The decisive gate is `test_minimax_music3_ar_real` — the +full-scale bf16 companion against the committed oracle goldens — run with the +device arm, and it needs the 28.5 GB checkpoint. It has NOT been run, and until +it has, `AGENTS.md`'s "establish the declared token-exact gate before you accept +a performance result" forbids quoting a Thor A/B for this arm. The A/B is +therefore **blocked on correctness, not on the box**, and that is why this row +lands without one. + +The stake is measurable rather than speculative: the header of +`minimax_music3_ar.h` records that an **fp32** depth forward left 448 450 of +716 800 committed golden values beyond one bf16 ULP, at mean absolute error +2.65e-03. A seam that keeps f32 through two roundings per layer is a step in that +direction, and whether it lands inside the full-scale gate's tolerance is exactly +what has not been measured. + +#### One instrument defect, found inside this row, and it printed a clean green + +The mutation runner restored the tree with `tar x`, which sets mtimes **from the +archive** — i.e. from before the mutation. The restored source was therefore +OLDER than the object ninja had built from the mutated one, **ninja skipped the +rebuild, and the next run reported 35/35 cases and 508/508 assertions SUCCESS +while executing the MUTATED binary**. `git diff --name-only` could not see it +either, because after restoration the file matches `HEAD` byte for byte and a +`sha256sum` of the source says — correctly, and uselessly — that the source is +pristine. + +That is `.agents/verification.md`'s "a copied build directory rebuilds the +original sources" arriving from the other side, and it is the same shape as +§16.6a: **whenever an artifact is required to have changed, assert that it +changed rather than that its inputs did.** The runner now restores with `tar xm`, +touches every source, and rebuilds before declaring the mutation over. The +numbers in the table above were re-taken on a forced rebuild. + +### 19.4b The tolerance was fitted to ONE seed, and a review falsified it + +§19.4a placed the bound from a single draw of the reference weights. A fresh +review changed **nothing but the RNG seed** — same distribution, same geometry, +no defect — and the shipped `mean <= 4.0` **reds on three of six equally valid +draws**. A tolerance a redraw can fail is measuring the draw, not the arm. + +The correct arm, `test_minimax_music3_ar`, 8 heads of 8, three layers, the full +8-position schedule, batch 2, on a CPU `vt::Queue`: + +| seed | worst | mean | median | under the old bound | +|---|---:|---:|---:|---| +| `0x9E3779B9` | 110 | 2.095 | 1 | pass (the seed it was fitted to) | +| `0x2468ACE0` | 435 | 3.804 | 1 | pass | +| `0x00000001` | 1110 | 5.755 | 1 | **FAIL** | +| `0x13579BDF` | 3663 | 7.154 | 1 | **FAIL** | +| `0xDEADBEEF` | 7340 | 9.904 | 1 | **FAIL** | +| `0x51ED2701` | 939 | 3.103 | 1 | pass | + +**`worst` cannot discriminate and is no longer gated.** A *correct* arm reads +worst 7340 at `0xDEADBEEF`; the gate/up half swap reads 6641 and the wrong +attention scale 6865. Any `worst` bound loose enough to admit a correct +implementation admits two structural defects, so a correct arm is *worse* on that +axis than two defects. It is reported, with a canary at 1e6 for a non-finite +value. + +**The metric itself had a defect, found by the sixth seed.** A reference value of +exactly zero has no ULP — bf16's spacing at zero is the denormal floor — so +`|got| / Bf16Ulp(0)` reads ~1e35 for an absolute difference of 1e-5. +`0x51ED2701` draws one such value in 1024 and read mean **8.8e32 with the arm +correct**. Zero references are now measured **absolutely** in their own bucket +(worst observed 8.30e-05 against a 1e-2 bound), and the two counts are asserted +to **sum**, so a defect cannot hide by growing the un-gated bucket. + +#### The battery, and why the median is the primary gate + +Five structural mutations, each run over all six seeds, on a clean tree, with the +compiler exit status, `git diff --stat` and the **binary** `sha256` printed for +every run and the source restored and verified after each: + +| mutation | mean range | median range | verdict | +|---|---:|---:|---| +| wrong attention scale (`1/sqrt(hidden)`) | 19.6 – 71.3 | 4 – 4 | RED 2 cases / 13 assertions | +| gate/up half swap | 52.5 – 158.8 | 12 – 15 | RED 2 / 14 | +| K/V cache row collision (`l*batch`) | 166.4 – 376.1 | 3 – 4 | RED 2 / 14 | +| dropped position embedding | 357.8 – 1487 | 76 – 90 | RED 2 / 15 | +| `q\|k\|v` merge order swapped | 400.0 – 1731 | 104 – 131 | RED 2 / 15 | + +**The median is exactly 1 bf16 ULP at every seed for the correct arm** — a +constant, not a distribution — because the deviation this arm carries is one +rounding-polarity tick per element and the seed only changes the tail. Every +defect is at least 3. The bound is **2**, and it cannot be moved by a redraw, +which is precisely the property the single-seed bound lacked. + +**The mean stays gated at 15, and its margin is honestly thin.** It is +tail-sensitive, so it catches a *sparse* defect that leaves the middle of the +distribution alone and the median would miss. Its window is +(9.904, 19.6): the correct arm's worst draw against the tightest defect, which is +the wrong attention scale at `0xDEADBEEF` — the same seed that produces the +correct arm's own worst mean. 15 is 1.51x above every correct draw measured and +1.31x below every defect draw measured. A seventh seed drawing a correct mean +above 15 is possible in a way that one drawing a median above 2 is not, and if +that happens the median is the gate that still holds. + +**This bound does not become obsolete.** §19.4 reason 3 records the verification: +the `vt::RmsNorm` term is two references legitimately disagreeing, not a defect +awaiting repair, so it does not close. #1322's surviving `vt::SiluAndMul` half +may narrow the band later; it is not landed, this row does not wait for it, and +nothing here is deferred against it. + +### 19.5 Reachability — the #1131 trap, named before it is fallen into + +[#1131](https://github.com/mudler/vllm.cpp/issues/1131) is this exact failure for +the DiT arm: its kernels and staging are gated, its **production switch** is not, +and setting `on_device = false` leaves every suite green. The reason it happened +is structural rather than careless — the CUDA arm needs a GPU that CI does not +have, so the natural gate is a unit test of the forward, and a unit test of the +forward is exactly what cannot see the switch. + +**The way out is that every op this arm uses has a CPU provider.** So the device +forward runs on a `vt::Queue` whose device is `kCPU`, with no GPU and no +checkpoint, and CI can drive it **through the production call site**: + +1. `Music3DepthStage` takes a `Music3DepthDeviceArm` (queue + staged weights). + Non-null selects `DepthDecoderAppendDevice`; null keeps the host loop. +2. `MiniMaxMusic3SpeechEngine` stages the arm through + `Music3SelectDepthArm`, on the **default** `--speech-device 1` — the same + switch `StageMusic3DitWeights` already rides. + + **This was written as an `if` in the engine, and a review proved that was the + trap it claims to avoid.** Deleting the whole + `if (queue_.device.type != kCPU) { … }` block — the only thing + `--speech-device 1` reaches — left `test_minimax_music3_ar` 35/35 · 508/508 + and `test_minimax_music3_speech` 9/9 · 223/223 SUCCESS. That is #1131's shape + reproduced by the change that names #1131 as its reason for existing, and it + is structural: on a CPU-only runner that condition can never be true, so no + branch written at that line is reachable by any gate CI owns. + + The rule now lives in `Music3SelectDepthArm`, which executes on **both** sides + of the condition and is therefore drivable from a CPU gate. Three outcomes, + and the third is the defect: a CPU queue stages nothing and returns a + disengaged arm; any other device stages and returns an engaged one, or + `StageMusic3DepthWeights` refuses by name because this build has no provider + for it; a non-CPU queue **quietly** taking the host loop is what must never + happen. Gutting the selector reds 1 case / 6 assertions. +3. The CI gate constructs the arm on a **CPU queue** and drives + `Music3DepthStage`, asserting (a) agreement with the host arm inside §19.4's + band, (b) identical drawn codes, and (c) that the device path was **taken** — + a counter, not an inference from the numbers, because the two arms agree + numerically by design. That last assertion is the one #1131 says is missing, + quoted in its own words: the gate "must assert the device path was TAKEN + (invocation count or resident dtype), not merely that outputs agree". +4. The reachability mutation deletes the production selection in a scratch copy + and the focused gate must go **RED**. It does: gutting `Music3SelectDepthArm` + reds 1 case / 6 assertions. + +**What is still NOT gated, stated rather than implied.** Deleting the engine's +two-line *call* to `Music3SelectDepthArm` leaves both suites green +(`test_minimax_music3_ar` 37/37 · 640/640, `test_minimax_music3_speech` +9/9 · 223/223), with the mutated `test_minimax_music3_speech` binary `sha256` +`77334986…` against the baseline `6bd42129…` so the run is not a stale-binary +artefact. Nothing here changes that, and nothing can on a CPU-only runner: the +engine needs the 28.5 GB checkpoint and a real device. The rule it calls is now +gated, its two components are gated, and the residual is the call itself. It is +owned by `MUSIC3-DEPTH-DEVICE` and tracked by +[#1131](https://github.com/mudler/vllm.cpp/issues/1131), listed under §19.7, and +it is the same residual the DiT block one screen below still carries in full. + +### 19.6 Gates and evidence this row owes + +| leg | where | what it proves | +|---|---|---| +| numeric agreement, CPU queue | `tests/vllm/models/test_minimax_music3_ar.cpp` | the composition, at reduced geometry, in CI, no GPU | +| composition + drawn codes | same | a dropped stage a tolerance cannot see | +| device path TAKEN | same | the #1131 hole | +| numeric agreement, CUDA | `thor:gpu0` under `rc` | the kernels the CPU provider does not exercise | +| stage A/B + wall | `thor:gpu0` under `rc` | the reason the row exists | +| WAV identity | `thor:gpu0` under `rc` | that it is inaudible in the product — **within** §19.4's band, so a HASH pair is not available here and an RMS/peak/max-abs comparison replaces it | + +**The A/B's own preconditions, which §16.6a paid for.** Two source trees, two +build dirs, both binaries' `sha256` printed and a hard failure when they are +**equal**; the checkpoint staged to local disk with `SRC_BYTES == DST_BYTES` +asserted; `uptime` on both sides; alternating pairs; the loudest pair kept. The +**call count is the control**: an arm that cannot move the counter cannot contain +the change. + +### 19.7 Owed, named here rather than discovered later + +Every item below is owned by row `MUSIC3-DEPTH-DEVICE` and names the issue that +tracks it, per `.agents/reachability.md` and `AGENTS.md` `## Nothing lands dead`. + +* **The engine's call to `Music3SelectDepthArm` is reachable but not gated** + ([#1131](https://github.com/mudler/vllm.cpp/issues/1131), row + `MUSIC3-DEPTH-DEVICE`). `--speech-device 1` reaches it and no CI gate can: + deleting the two-line call leaves `test_minimax_music3_ar` 37/37 · 640/640 and + `test_minimax_music3_speech` 9/9 · 223/223 green, because the engine needs the + 28.5 GB checkpoint and a real device. §19.5 carries the mutation and the binary + hashes. The *rule* it calls is gated on both sides of its condition, so what is + owed is the call, not the logic. It closes with the `thor:gpu0` legs in §19.6. +* **`scripts/check-fusion-consistency.py` is satisfied by a COMMENT** + ([#1351](https://github.com/mudler/vllm.cpp/issues/1351), row + `MUSIC3-DEPTH-DEVICE`). Replacing the `layers::UnquantizedMlpGateUpMethod` call + in `minimax_music3_depth_device.cpp` with an inline hand-rolled + `vt::MatmulBT` + `vt::SiluAndMul` path, while leaving the two comment mentions + in place — the `#include`'s trailing `// layers::UnquantizedMlpGateUpMethod` + and the `gate_up` field note — leaves the checker **green**, and the numbers + bit-identical so nothing else sees it either. Verified two ways: a fresh + reviewer ran the checker end to end at `rc=0`, and calling the checker's own + functions gives `uses_merged_gemm_seam = True` for the hand-rolled variant with + comments kept and `False` only once comments are stripped. `Check 2` runs + `_MERGED_GEMM_SEAM` over `path.read_text()`; the file's only comment-aware + helper, `allowlisted_names`, strips `#` comments from the **allowlist file** + rather than from the scanned source. So the checker can no longer detect a + regression to a hand-rolled path in this TU. **Reported rather than fixed + here:** a checker change needs its own spec, a red-before test and green-after + evidence per `AGENTS.md` `## Changing the rules or a checker`, and widening the + regex is exactly the move that section forbids. Renaming `MlpGateUp` to + `MlpGateUpXX` is *not* a detection gap — `MlpGateUp[A-Za-z]*Method` matches it + by design, to cover `UnquantizedMlpGateUpGeluMethod`. +* **The depth K/V cache's pooling is unmeasured** + ([#1309](https://github.com/mudler/vllm.cpp/issues/1309), row + `MUSIC3-DEPTH-DEVICE`). It now draws from the device pool through `DBuf` + instead of `backend.Alloc`/`Free`, which at the shipped 4-layer batch-2 + geometry removes 16 `cudaMalloc` and 16 synchronizing `cudaFree` per frame. The + change is correct by construction — `ReleaseShared` returns the block to the + pool it came from — but **no speed number is quoted for it**, because this box + has no GPU and the A/B is blocked on correctness. Measure it with the §19.6 + `thor:gpu0` legs, not before. +* The **audio heads, the CFG mix, the top-k draw and the fed-back projection** + stay on the host. They are ~1.6 % of the stage today; they become the stage's + remainder once the forward moves, and that is the next thing to measure rather + than to assume. +* The **host arm's f32 weight containers** are unchanged (§19.2a). §16.7's lever 1 + — bf16 host storage, bit-identical on the safetensors lineage and **not** on + the GGUF Q4_K one — is still open and is still not this row. +* The **condition mix** is still host-side (§14.5). It runs once per window, not + once per step. +* **Why Thor's stage ratios exceed x86's** is still a hypothesis (§16.6b) and + this row does not settle it. + +### 19.8 Stop conditions + +Stop and report rather than widening scope if: the CUDA arm's disagreement with +the host arm exceeds §19.4's band and the cause is not one of that section's +three; `vt::AttentionCross` refuses this geometry; the staged bf16 weights do not +fit beside the 8.6 B language model on the measurement box; or the A/B's +`ARMS_DIFFER` guard fires. diff --git a/CMakeLists.txt b/CMakeLists.txt index 71bdfeb37..79a85c242 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -781,6 +781,7 @@ add_library(vllm STATIC src/vllm/model_executor/models/minimax_music3_ar.cpp src/vllm/model_executor/models/minimax_music3_acoustic.cpp src/vllm/model_executor/models/minimax_music3_device.cpp + src/vllm/model_executor/models/minimax_music3_depth_device.cpp src/vllm/model_executor/models/minimax_music3_llm.cpp src/vllm/model_executor/models/minimax_music3_speech.cpp src/vllm/model_executor/models/gpt2.cpp diff --git a/docs/USAGE.md b/docs/USAGE.md index 22121d66b..3e19debd0 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -2251,7 +2251,7 @@ direction that matters. | 8.6B `Qwen3ForCausalLM` (prefill + every decode step, its paged KV) | **device** | | guided-logit pipeline, top-k draw, frame feedback embedding | host (two 200 000-wide rows per step; not the cost) | | **2.4B fp32 flow-matching DiT** (every denoise step, both CFG branches) | **device**, weights staged ONCE | -| 0.646B RVQ depth decoder (7 steps per frame) | **host**, scalar loops | +| **0.646B RVQ depth decoder** (8 appends per frame) | **device**, weights staged ONCE at **bf16** | | condition mix (once per window), scheduler, CFG mix, Euler step | **host** | | DAC Flow-VAE vocoder (`Conv1d` / `ConvTranspose1d`) | **host**, scalar loops | @@ -2272,12 +2272,10 @@ this arm mirrors that rather than buying speed with a narrower dtype. The remaining stages do not move, for two different reasons, and both are owed rather than hidden: -* the depth decoder and the condition mix are host `std::vector` - reference loops under `-ffp-contract=off`, and they run at - `ArCompute::kBFloat16` — every op's *result* is rounded to bf16, which is what - upstream stores. Routing them through an f32 GEMM would silently drop that - rounding, so mirroring them needs bf16 storage, which is a dtype decision with - its own numeric evidence rather than a transcription; +* the condition mix is a host `std::vector` reference loop under + `-ffp-contract=off` running at `ArCompute::kBFloat16`. It also runs **once per + window** rather than once per step, so it is outside the per-step loop + entirely; * the vocoder needs `ConvTranspose1d`, and **`vt` has no such op at all** — the 1-D convolutions it does have (`vt::DepthwiseConv1d`, `vt::CausalConv1dFwd`) are depthwise or causal-with-state, and `vt::Conv2d` and `vt::DepthwiseConv1d` @@ -2285,6 +2283,58 @@ rather than hidden: this stage would need, so it is named here rather than hand-rolled outside the seam. +The **depth decoder** reaches the device the same way, and its dtype is the one +thing about it worth knowing. Upstream's `MiniMaxMusic3RVQDepthDecoder` declares +**no dtype at all** — no `dtype` parameter, no `torch.float32` literal, no +`.float()` call — so it inherits whatever `load_components(dtype=...)` resolves, +and that is **bf16** for this checkpoint. The arm therefore stages its weights at +bf16 and keeps every activation at bf16 with f32 accumulation, which is +`vt::MatmulBT`'s own contract. The narrowing is **lossless**: the loader already +rounds every AR-half tensor through bf16 into an f32 carrier, so the device copy +holds exactly the values the host arm holds, in half the bytes. + +It runs on five shared ops with **no new kernel** (`MatmulBT`, `RmsNorm`, +`AttentionCross`, `SiluAndMul`, `Add`), with the MLP's gate/up pair routed +through the shared merged-GEMM seam. What does **not** move with it, and is owed +rather than hidden: the audio heads, the CFG mix, the top-k draw and the fed-back +projection row, which together are ~1.6 % of the stage. + +**Its numbers are not identical to the host arm's, and the difference is +measured rather than assumed.** Against the host reference, over **six seeds** of +the gate's reduced geometry, the device arm reads a median of exactly **1 bf16 +ULP** at every seed, means of 2.095 to 9.904 and worst-case values of 110 to +7340. The gate bounds the median at 2 and the mean at 15; it does **not** bound +the worst case, because the worst case cannot tell a correct arm from a broken +one — a correct arm reads 7340 on the seed where a swapped gate/up half reads +6641. An earlier revision of this document quoted one seed's 110 and 2.095 as +though they were the arm's deviation; they are one draw of it. + +Almost all of that deviation — the composed stage goes to **zero, +bit-identical**, once the two are aligned — is one rounding per element, from two +places, and the two are **not** the same kind of thing: + +- `vt::SiluAndMul` computes the whole gated expression in f32 where a bf16 torch + module narrows `silu(gate)` before multiplying by `up`. That one is a genuine + gap in the shared seam and is tracked as its own issue. +- `vt::RmsNorm` keeps f32 across the weight multiply, and **that is correct**. + vLLM's own RMSNorm does exactly the same on both its CPU and its CUDA path, and + upstream reverted the change that would have made it multiply in the weight's + dtype. The Music3 *host* arm rounds twice because it mirrors the `diffusers` + module this model is; the device arm rounds once because it mirrors vLLM. Two + references disagree here and neither side is defective, so this term will not + go away. An earlier revision of this document called it a shortcoming of the + shared op. + +Until the difference is settled against the oracle, **no throughput number is +quoted for this arm** and the full-scale gate against the committed oracle +goldens has not been run with it. + +If the build has no provider for the device you asked for, the depth arm +**refuses by name at staging time**, naming the op and the device, rather than +falling back to the host loop — a silent fallback would be a large slowdown +wearing a correct answer. `--speech-device 0` keeps the host reference arm and +stages nothing. + Because the host stages are unchanged — and because `--speech-device 0` takes the same `DitForward` it always did, source byte for source byte — the CPU arm is **bit-identical** to the one every Music3 correctness gate was taken on. The diff --git a/include/vllm/model_executor/models/minimax_music3_depth_device.h b/include/vllm/model_executor/models/minimax_music3_depth_device.h new file mode 100644 index 000000000..5acaaa7d0 --- /dev/null +++ b/include/vllm/model_executor/models/minimax_music3_depth_device.h @@ -0,0 +1,259 @@ +// MiniMax-Music3 — the DEVICE-RESIDENT RVQ depth decoder (#672, #1309, spec §19). +// +// `DepthDecoderAppend` (minimax_music3_ar.cpp) is the portable reference: host +// `std::vector` throughout, one sequential `double` accumulator per +// output element. It is what every Music3 gate was taken on, it is NOT changed +// by this header, and it stays the CPU arm. This is an ADDITIONAL entry point. +// +// ─── WHY THIS EXISTS ───────────────────────────────────────────────────────── +// +// Spec §19.1, measured on `thor:gpu0` at `678fc672c`: `ar.depth_forward` is +// 78.1 s of a 161 s run — 48.4 %, the largest single term. A 0.646B decoder +// therefore costs 6.3x the 8.6B language model beside it, and the reason is +// which processor each runs on. §16.4 measured the host kernel at ~4 bytes of +// f32 weight traffic per multiply-accumulate — 2.28 GB per call, 8 calls a +// frame — and §16.6b measured 96.93 ms per call, i.e. ~23.5 GB/s across Thor's +// 14 cores. This arm moves that sweep to the device and halves it. +// +// ─── WHAT IS PORTED, AND ONTO WHICH SHARED OP ──────────────────────────────── +// NO new kernel. Every op below already carries a CPU AND a CUDA provider. +// +// RmsNorm -> vt::RmsNorm +// LinearNoBias -> vt::MatmulBT +// CausalAttentionStep -> vt::AttentionCross (bias = nullptr) +// silu(gate) * up -> vt::SiluAndMul +// residual adds -> vt::Add +// +// THE ATTENTION IS CAUSAL UPSTREAM AND `vt::AttentionCross` IS STILL THE RIGHT +// OP, by an identity rather than a shortcut. This call presents ONE query row +// against a cache of `seq` positions that are all at or before it, so a causal +// mask masks nothing. `vt::Attention` would apply a mask keyed on a query index +// this call does not have. Upstream's `dispatch_attention_fn(..., is_causal=True)` +// (minimax_music3_rvq_depth_decoder.py:43-50) runs whole sequences; §16.3's +// identity is why ours runs one row, and that identity is gated bitwise. +// +// ─── bf16 STORAGE, f32 ACCUMULATION — THE DECISION §14.5 LEFT OWED ─────────── +// +// Spec §19.2. The oracle settles this by declaring NOTHING: +// `MiniMaxMusic3RVQDepthDecoder` takes no `dtype` parameter and contains no +// `torch.float32` literal and no `.float()` call +// (minimax_music3_rvq_depth_decoder.py:101-125); its single cast is the +// DOWN-cast `:51` `hidden_states.flatten(2, 3).to(query.dtype)`. The dtype is +// imposed by `load_components(dtype=...)`, and tools/oracle/music3_oracle.py +// resolves `rvq_depth_decoder: torch.bfloat16` under BOTH its policies. That is +// the §2.1 invariant `dtype(language_model) == dtype(rvq_depth_decoder) == +// dtype(condition_encoder)`. +// +// `vt::MatmulBT` has carried exactly that contract all along — "a/b bf16 (or +// f32), out f32 or bf16, f32 accumulation" (vt/ops.h) — so §14.5's objection, +// that an *f32* `vt::MatmulBT` would drop the bf16 rounding every gated number +// was taken with, is answered by using the *bf16* one. The CUDA provider refuses +// a mixed f32-activation x bf16-weight combo by name, so this arm is bf16 on +// both operands or it is not bf16 at all. +// +// THE NARROWING IS LOSSLESS, WHICH IS WHY IT IS SAFE TO CALL A MIRROR RATHER +// THAN A CHANGE. `AtRuntimeDtype` (minimax_music3_llm.cpp) already rounds every +// AR-half tensor through bf16 into an f32 carrier, and `Store(..., kBFloat16)` +// does the same to every activation the host arm hands across this boundary. So +// every value staged or uploaded here is ALREADY exactly bf16-representable and +// `vt::F32ToBF16` on it is exact. The host arm's f32 CONTAINERS are what move +// twice the oracle's bytes (spec §19.2a); this arm's do not. +// +// ─── NUMERICS: NOT BIT-IDENTICAL, AND SAID BEFORE THE CODE ─────────────────── +// +// Spec §19.4. §16 could claim bitwise identity because a causal identity is +// exact; this row changes the machine, and it does NOT claim it. Three +// independent reasons, each sufficient alone: +// +// 1. THE ACCUMULATOR NARROWS. The host keeps a sequential `double` per output +// element; `vt::MatmulBT` accumulates in f32 (CUBLAS_COMPUTE_32F). Over +// in_dim 4096 and 6144 that is real. Note the DIRECTION: torch's own bf16 +// matmul accumulates in f32, so on this axis the device arm is the CLOSER +// mirror of upstream, not the looser one. +// 2. THE REDUCTION RE-ASSOCIATES. cuBLASLt splits K by an algorithm this file +// does not choose, and `vt::AttentionCross`'s CUDA kernel uses an online +// softmax recurrence where the host and the CPU provider use an explicit +// three-pass max/exp/normalize. Those two are not bit-identical to each +// other either. +// 3. THE TWO ARMS NORMALIZE AGAINST DIFFERENT REFERENCES, AND BOTH ARE RIGHT. +// The Music3 HOST arm's own RmsNorm (minimax_music3_ar.cpp) mirrors the +// diffusers module this model IS: `normalization.py::RMSNorm.forward` casts +// back to the weight dtype at `:560` before the affine multiply at `:561`, +// so it rounds TWICE, and the decoder constructs exactly that class +// (minimax_music3_rvq_depth_decoder.py:78,80,122). The device arm routes +// through the SHARED `vt::RmsNorm`, which mirrors vLLM — and vLLM keeps f32 +// across the weight multiply and rounds ONCE. Verified at the parity pin +// `555967922`: `csrc/cpu/layernorm.cpp` computes +// `fp32_out = fp32_x * fp32_s_variance * fp32_w`, and +// `csrc/libtorch_stable/layernorm_kernels.cu:93` computes +// `static_cast(x * s_variance * w)`. Upstream tried the +// weight-dtype multiply (vllm#42379) and REVERTED it (vllm#46070, an +// ancestor of the pin). +// +// So this is NOT the shared op diverging from its reference, and an earlier +// revision of this comment said it was. AGENTS.md makes vLLM the only +// reference wherever it implements the behaviour, and it implements +// RMSNorm, so `vt::RmsNorm` is correct against it. What differs is which +// reference each arm answers to. WHICH ONE IS RIGHT FOR THIS MODEL is a +// question the diffusers oracle settles through +// `test_minimax_music3_ar_real`, and §19.6 records that as owed. +// +// The gate is therefore a tolerance in bf16 ULPs of the reference value, and it +// asserts its own teeth. A tolerance cannot see a DROPPED STAGE, so it is not +// the only gate: the composed schedule's drawn codes are compared too. +#pragma once + +#include +#include +#include + +#include "vllm/model_executor/models/minimax_music3_ar.h" +#include "vllm/model_executor/models/qwen3_5_weights.h" // OwnedTensor, for the merged gate_up +#include "vt/device.h" +#include "vt/ops.h" + +namespace vllm { +namespace models { +namespace music3 { + +// One decoder layer's weights, resident on the queue's device, bf16. +// +// TWO MERGES, because AGENTS.md routes mergeable projections through the merged +// seam and because they cost nothing numerically — each output element remains +// its own dot product, so no reduction moves: +// +// * `qkv` is `to_q | to_k | to_v` stacked as one [3H, H]. One `vt::MatmulBT` +// sweeps the weight once for all three where the host arm sweeps three +// times, and the three results are read back by free re-view. +// * `gate_up` is `gate_proj | up_proj` stacked as one [2I, H], and it is +// consumed through `layers::UnquantizedMlpGateUpMethod` — the shared +// merged-GEMM seam, not a hand-rolled equivalent. The merge is REQUIRED +// rather than merely useful: `vt::SiluAndMul` consumes ONE [T, 2D] buffer as +// `silu(x[:, :D]) * x[:, D:]`, and upstream's +// `silu(gate_proj(x)) * up_proj(x)` puts gate first. So NO half swap — +// unlike §14.3's DiT, where the assignment was the other way round and +// needed a stage-time exchange. +struct Music3DepthDeviceLayer { + vt::Tensor input_layernorm; // [H] + vt::Tensor post_attention_layernorm; // [H] + vt::Tensor qkv; // [3H, H] MERGED + vt::Tensor to_out; // [H, H] + // [2I, H] MERGED, and an `OwnedTensor` rather than a staged `vt::Tensor` + // because it is consumed through `layers::UnquantizedMlpGateUpMethod`, which + // is the shared merged-GEMM seam AGENTS.md routes mergeable MLP projections + // through. That method IS this arm's MLP byte for byte — one `vt::MatmulBT` + // over the merged [2I, H] operand then `vt::SiluAndMul` — so routing through + // it costs nothing and puts the fused-kernel scheme choice (a quantized arm + // may fuse the pair into one Marlin GEMM) in its one home. `ResidentWeight` + // owns the residency: it ALIASES on a CPU queue and uploads once on a device. + OwnedTensor gate_up; + vt::Tensor down_proj; // [H, I] +}; + +// The depth decoder staged ONCE onto a device, with the storage that owns it. +// +// STAGED ONCE IS THE POINT, as it is for the DiT: a 4 s clip runs this decoder +// 808 times, so a per-call upload of the weights would cost more than the +// compute it enables. +// +// `projection`, `audio_embeddings` and `audio_heads` are NOT here. They stay on +// the host in this row, deliberately: §15's profile puts `ar.depth_projection` +// at 1.307 s against the forward's 78.316 s, so they are ~1.6 % of the stage. +// Spec §19.7 carries them as owed rather than leaving them to be discovered. +struct Music3DepthDeviceWeights { + vt::Tensor pos_embedding; // [max_position_embeddings, H] + vt::Tensor norm; // [H] + std::vector layers; + + std::vector> storage; + + bool staged() const { return !layers.empty(); } +}; + +// Upload the depth decoder to `queue`'s device, once, narrowing to bf16. +// +// `release_host` empties each source vector as it is uploaded, for the same +// unified-memory reason `StageMusic3DitWeights` has it: on Jetson Thor host and +// device draw on one pool, so holding both copies is a real peak. Pass true from +// a serving path and false from a gate that compares the two arms. +// +// Throws, naming the tensor, if a weight is mis-sized for `config`; and throws +// naming the op and the device if the device has no provider for one of the ops +// the forward needs — a refusal at stage time rather than four layers into the +// first of 808 calls. +Music3DepthDeviceWeights StageMusic3DepthWeights(vt::Queue& queue, + const DepthDecoderConfig& config, + DepthDecoderWeights& weights, + bool release_host); + +// The device twin of `DepthDecoderCache`: the K and V of every position already +// fed, for `batch` INDEPENDENT rows sharing one weight set. +// +// ONE CONTIGUOUS BLOCK PER (layer, row) of [max_position_embeddings, H], sized +// once at first use. `vt::AttentionCross` requires contiguous operands, so a +// row's history must be contiguous ACROSS positions — which fixes this layout +// and is why the batched K/V projection is copied into the cache rather than +// written there directly by the GEMM. The copy is H bf16 values per row per +// tensor (8 KB at the shipped geometry), against a 2.28 GB weight sweep. +struct Music3DepthDeviceCache { + int64_t batch = 0; + int64_t hidden = 0; + int64_t layers = 0; + int64_t positions = 0; + int64_t capacity = 0; + // [layer * batch + row] -> [capacity, hidden] bf16, device-resident. + std::vector keys; + std::vector values; + std::vector> storage; +}; + +// How many times `DepthDecoderAppendDevice` has run in this process. +// +// #1131 IS WHY THIS EXISTS. The DiT device arm's kernels and staging are gated +// and its production SWITCH is not: setting `on_device = false` leaves every +// suite green, because the two arms agree numerically BY DESIGN and no gate ever +// asked which one ran. `.agents/verification.md` requires the gate to assert the +// device path was TAKEN — "invocation count or resident dtype", not an inference +// from the numbers. This counter is that assertion's instrument, and it is the +// reason a reachability mutation on the production selection goes RED here. +uint64_t Music3DepthDeviceForwardCount(); + +// #1131's OTHER instrument, and this row took only the first until a review said +// so. The words are #1131's own: a gate "must assert the device path was TAKEN +// (invocation count OR RESIDENT DTYPE), not merely that outputs agree". +// +// A bit per `vt::DType`, OR-ed over every buffer `DepthDecoderAppendDevice` +// makes resident — the activations, the merged QKV block, the MLP result, the +// downloaded output and the K/V cache — accumulated across every call in this +// process. It reads back what the forward ACTUALLY allocated rather than what +// this header says it should, which is the whole point: AGENTS.md's "a token +// gate cannot detect a dtype that is too WIDE" applies with full force to the +// tolerance gate above, and a review proved it — widening one activation buffer +// to `kF32` left the ULP band, the drawn codes and all 35 cases green while the +// path moved twice the bytes. That is §19.2a's own finding about the HOST arm, +// arriving inside the arm that exists to fix it. +// +// The mask is monotone and never reset, so a single widened buffer anywhere in +// the process is visible for the rest of it. +uint64_t Music3DepthDeviceResidentDtypes(); + +// ONE depth position, appended to `cache`, for `batch` rows at once — the device +// twin of `DepthDecoderAppend`, with the same inputs, the same outputs and the +// same refusals. +// +// `inputs_embeds` is [batch, hidden_size] on the HOST carrying bf16-exact +// values; the return is [batch, hidden_size] on the HOST, the post-`norm` hidden +// state of the appended position for each row — the only row the generation +// schedule reads (encoders.py:131-132). +// +// The host<->device boundary is the ARGUMENTS ONLY: one upload of [batch, H] in +// and one download of [batch, H] out per call. The weights and the K/V history +// never cross it. +std::vector DepthDecoderAppendDevice(vt::Queue& queue, const DepthDecoderConfig& config, + const Music3DepthDeviceWeights& weights, + const std::vector& inputs_embeds, + int64_t batch, Music3DepthDeviceCache* cache); + +} // namespace music3 +} // namespace models +} // namespace vllm diff --git a/include/vllm/model_executor/models/minimax_music3_llm.h b/include/vllm/model_executor/models/minimax_music3_llm.h index 981221511..5669acc64 100644 --- a/include/vllm/model_executor/models/minimax_music3_llm.h +++ b/include/vllm/model_executor/models/minimax_music3_llm.h @@ -76,6 +76,7 @@ #include #include "vllm/model_executor/models/minimax_music3_ar.h" +#include "vllm/model_executor/models/minimax_music3_depth_device.h" #include "vllm/model_executor/models/minimax_music3_loader.h" #include "vllm/model_executor/models/qwen3.h" #include "vllm/tokenizer/tokenizer.h" @@ -265,6 +266,47 @@ struct Music3ArResult { bool stopped_on_end_token = false; }; +// The depth decoder's DEVICE arm, selected per call (#1309, spec §19). +// +// Both fields or neither. One alone is a caller that thinks it asked for the +// device arm and did not, so it is REFUSED rather than silently ignored — the +// same shape, and the same reason, as `Music3DenoiseDeviceArm`. +// +// A default-constructed arm keeps the host loop, so every existing caller and +// every existing gate is unchanged by construction. +struct Music3DepthDeviceArm { + vt::Queue* queue = nullptr; + const Music3DepthDeviceWeights* depth = nullptr; + bool engaged() const { return queue != nullptr && depth != nullptr; } + bool half_set() const { return (queue != nullptr) != (depth != nullptr); } +}; + +// THE PRODUCTION SELECTION of that arm, as a function rather than as an `if` in +// the engine (#1309, [#1131](https://github.com/mudler/vllm.cpp/issues/1131), +// spec §19.5). +// +// WHY IT IS A FUNCTION. The engine's condition is `queue_.device.type != kCPU`, +// and that condition is FALSE on every runner CI owns, so an `if` written there +// is the one line a CPU-only gate can never enter — which is #1131 exactly, and +// #1131 is the reason this row exists. The same function runs on both sides of +// the condition, so a gate CAN enter it: with a CPU queue, and with a fabricated +// non-CPU one. +// +// THREE OUTCOMES, AND THE THIRD IS THE DEFECT. A CPU queue stages nothing and +// returns an arm that is not engaged, so the caller keeps the host reference loop +// every Music3 gate was taken on. Any other device stages the decoder into +// `*staged` and returns an ENGAGED arm — or, on a build or a box that has no +// provider for it, `StageMusic3DepthWeights` REFUSES by name and that refusal +// propagates. What must never happen is a non-CPU queue quietly taking the host +// arm, and that is what `test_minimax_music3_ar` asserts over every non-CPU +// `vt::DeviceType`. +// +// `staged` outlives the returned arm's use or the arm dangles; the engine holds +// both in the same scope. Throws if it is null. +Music3DepthDeviceArm Music3SelectDepthArm(vt::Queue& queue, const DepthDecoderConfig& config, + DepthDecoderWeights& weights, bool release_host, + Music3DepthDeviceWeights* staged); + // `MiniMaxMusic3SemanticGenerationStep.__call__` (encoders.py:299-353). // // `prompt_ids` is the CONDITIONAL row; the unconditional row is derived with @@ -278,7 +320,8 @@ Music3ArResult Music3GenerateFrameHiddens(const std::vector& prompt_ids int64_t max_frames, const Music3ArWeights& weights, const Music3CodeSampler& sampler, - vt::Queue& queue); + vt::Queue& queue, + const Music3DepthDeviceArm& device_arm = {}); // One frame's DEPTH stage, with the language model's own hidden states SUPPLIED // rather than produced. `Music3GenerateFrameHiddens` calls it once per frame, @@ -300,7 +343,8 @@ std::vector Music3DepthStage(const std::vector& last_hidden_condit const std::vector& last_hidden_unconditional, int64_t frame_index, const Music3ArWeights& weights, const Music3CodeSampler& sampler, - std::vector* out_frame_codes); + std::vector* out_frame_codes, + const Music3DepthDeviceArm& device_arm = {}); } // namespace music3 } // namespace models diff --git a/src/vllm/model_executor/models/minimax_music3_depth_device.cpp b/src/vllm/model_executor/models/minimax_music3_depth_device.cpp new file mode 100644 index 000000000..8ab4f1269 --- /dev/null +++ b/src/vllm/model_executor/models/minimax_music3_depth_device.cpp @@ -0,0 +1,419 @@ +// MiniMax-Music3 — the DEVICE-RESIDENT RVQ depth decoder. See +// minimax_music3_depth_device.h for what is ported onto which shared op, why the +// dtype is bf16 and why the narrowing is lossless, and the three named reasons +// this arm is close to but not bit-identical to the host reference. +// +// ─── THE ONE STRUCTURAL DECISION IN THIS FILE ──────────────────────────────── +// +// `vt::AttentionCross` requires CONTIGUOUS operands, so a row's key history must +// be contiguous ACROSS positions. That fixes the cache as one [capacity, H] +// block per (layer, row) and means the batched K/V projection — which produces +// [batch, 3H] with the two rows interleaved — cannot be written into the cache +// by the GEMM itself. It is copied in, H bf16 values per row per tensor. +// +// The alternative was a per-row K/V GEMM writing straight into the slot, and it +// is the wrong trade by an order of magnitude: it would sweep the [H, H] weight +// once PER ROW instead of once per call, which is exactly the weight-streaming +// cost §16.4 measured and this arm exists to remove. An 8 KB copy against a +// 2.28 GB sweep is not a trade at all. +#include "vllm/model_executor/models/minimax_music3_depth_device.h" + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "vllm/model_executor/models/dense_device_glue.h" +#include "vllm/model_executor/layers/linear.h" // layers::UnquantizedMlpGateUpMethod +#include "vt/backend.h" +#include "vt/dtype.h" +#include "vt/op_provider.h" // GetOp — the stage-time provider refusal +#include "vt/ops.h" + +namespace vllm { +namespace models { +namespace music3 { + +namespace { + +using dense_attn::DBuf; +using dense_attn::Dev; +using dense_attn::MakeTensor; +using vt::DType; +using vt::Tensor; + +[[noreturn]] void Fail(const std::string& message) { throw std::runtime_error(message); } + +// See `Music3DepthDeviceForwardCount`. Relaxed because this is an instrument +// read after a synchronizing call, never a synchronization primitive itself. +std::atomic g_forward_count{0}; + +// See `Music3DepthDeviceResidentDtypes`. Same ordering argument as the counter +// above: an instrument, not a synchronization primitive. +std::atomic g_resident_dtypes{0}; + +// Record the dtype of a buffer the forward made resident. `DType` is a `uint8_t` +// enum of 20 values, so one bit each fits a `uint64_t` with room to spare, and +// the static_assert makes adding the 65th dtype a build error rather than a +// silently truncated instrument. +constexpr uint64_t DtypeBit(DType dt) { + return uint64_t{1} << static_cast(dt); +} +static_assert(static_cast(DType::kMXFP4) < 64, + "the resident-dtype mask has one bit per DType and DType outgrew 64"); + +void RequireStageSize(const std::vector& values, int64_t expected, const char* what) { + if (static_cast(values.size()) != expected) { + Fail(std::string("MiniMax-Music3 depth stage: ") + what + " is " + + std::to_string(values.size()) + " values, expected " + std::to_string(expected)); + } +} + +// A bf16 device view over a fresh block, owned by `storage`, built from one or +// more f32 host sources CONCATENATED in order — which is how the two merged +// projections are assembled without a second host buffer. +// +// THE NARROWING IS EXACT AND THAT IS LOAD-BEARING, not a rounding we tolerate. +// `AtRuntimeDtype` already rounded every AR-half weight through bf16 into its +// f32 carrier, so `F32ToBF16` here is the inverse of the widening the loader +// did, value for value. See the header. +Tensor UploadBf16(vt::Backend& backend, vt::Queue& queue, + const std::vector*>& sources, + const std::vector& shape, bool release, + std::vector>* storage) { + int64_t numel = 1; + for (int64_t s : shape) numel *= s; + std::vector packed(static_cast(numel)); + size_t at = 0; + for (std::vector* src : sources) { + for (float v : *src) packed[at++] = vt::F32ToBF16(v); + } + if (at != packed.size()) { + Fail("MiniMax-Music3 depth stage: staged " + std::to_string(at) + " values into a " + + std::to_string(packed.size()) + "-value tensor"); + } + const size_t bytes = packed.size() * sizeof(uint16_t); + void* p = backend.Alloc(bytes); + std::shared_ptr owner(p, [&backend](void* q) { backend.Free(q); }); + backend.Copy(queue, p, packed.data(), bytes); + // The copy must have LANDED before `packed` leaves scope. On a CPU queue this + // is a memcpy and the sync is free; on CUDA the source is pageable host + // memory, so returning under an unsynchronized async copy is exactly the + // use-after-free that reads as a plausible-looking wrong tensor. + backend.Synchronize(queue); + if (release) { + for (std::vector* src : sources) std::vector().swap(*src); + } + storage->push_back(std::move(owner)); + return MakeTensor(p, DType::kBF16, queue.device, shape); +} + +// The same lossless f32 -> bf16 concatenation as `UploadBf16`, but left on the +// HOST as an `OwnedTensor` so the shared merged-GEMM seam owns its residency. +// `nk = true` is the torch Linear [N=out, K=in] orientation `vt::MatmulBT` +// consumes, which is the orientation these weights already have. +OwnedTensor PackBf16Owned(const std::vector*>& sources, + const std::vector& shape, bool release) { + int64_t numel = 1; + for (int64_t s : shape) numel *= s; + std::vector raw(static_cast(numel) * sizeof(uint16_t)); + auto* packed = reinterpret_cast(raw.data()); + size_t at = 0; + for (std::vector* src : sources) { + for (float v : *src) packed[at++] = vt::F32ToBF16(v); + } + if (at != static_cast(numel)) { + Fail("MiniMax-Music3 depth stage: packed " + std::to_string(at) + " values into a " + + std::to_string(numel) + "-value tensor"); + } + if (release) { + for (std::vector* src : sources) std::vector().swap(*src); + } + OwnedTensor out; + out.bytes = OwnedBytes(std::move(raw)); + out.dtype = DType::kBF16; + out.rank = static_cast(shape.size()); + for (size_t i = 0; i < shape.size(); ++i) out.shape[i] = shape[i]; + out.nk = true; + return out; +} + +// A contiguous sub-view of a bf16 [rows, width] buffer starting at `row`. +Tensor RowView(const Tensor& base, int64_t row, const std::vector& shape) { + auto* p = static_cast(base.data) + row * base.shape[base.rank - 1]; + return MakeTensor(p, DType::kBF16, base.device, shape); +} + +} // namespace + +uint64_t Music3DepthDeviceForwardCount() { return g_forward_count.load(std::memory_order_relaxed); } + +uint64_t Music3DepthDeviceResidentDtypes() { + return g_resident_dtypes.load(std::memory_order_relaxed); +} + +Music3DepthDeviceWeights StageMusic3DepthWeights(vt::Queue& queue, + const DepthDecoderConfig& config, + DepthDecoderWeights& weights, + bool release_host) { + const int64_t hidden = config.hidden_size; + const int64_t inter = config.intermediate_size; + const int64_t heads = config.num_attention_heads; + if (heads <= 0 || hidden % heads != 0) { + Fail("MiniMax-Music3 depth stage: hidden_size " + std::to_string(hidden) + + " is not divisible by num_attention_heads " + std::to_string(heads)); + } + if (static_cast(weights.layers.size()) != config.num_layers) { + Fail("MiniMax-Music3 depth stage: the weights carry " + + std::to_string(weights.layers.size()) + " layers, the config declares " + + std::to_string(config.num_layers)); + } + + // REFUSE UP FRONT, before 1.3 GB moves. `vt::GetOp` throws naming the op and + // the device, so a backend without (say) a cross-attention provider is a + // one-line refusal at stage time rather than a failure four layers into the + // first of 808 calls. Every op the forward below calls is listed. + for (vt::OpId op : {vt::OpId::kMatmulBT, vt::OpId::kAdd, vt::OpId::kRmsNorm, + vt::OpId::kSiluAndMul, vt::OpId::kAttentionCross}) { + (void)vt::GetOp(op, queue.device.type); + } + + // EVERY size is checked BEFORE anything is uploaded, and therefore before + // `release_host` destroys anything. A size error found after nine layers had + // been released would be a correct refusal that had already consumed the + // caller's weights. + RequireStageSize(weights.pos_embedding, config.max_position_embeddings * hidden, + "pos_embedding"); + RequireStageSize(weights.norm, hidden, "norm"); + for (size_t l = 0; l < weights.layers.size(); ++l) { + const DepthDecoderLayerWeights& layer = weights.layers[l]; + const std::string at = "layer " + std::to_string(l) + " "; + RequireStageSize(layer.input_layernorm, hidden, (at + "input_layernorm").c_str()); + RequireStageSize(layer.post_attention_layernorm, hidden, + (at + "post_attention_layernorm").c_str()); + RequireStageSize(layer.to_q, hidden * hidden, (at + "to_q").c_str()); + RequireStageSize(layer.to_k, hidden * hidden, (at + "to_k").c_str()); + RequireStageSize(layer.to_v, hidden * hidden, (at + "to_v").c_str()); + RequireStageSize(layer.to_out, hidden * hidden, (at + "to_out").c_str()); + RequireStageSize(layer.gate_proj, inter * hidden, (at + "gate_proj").c_str()); + RequireStageSize(layer.up_proj, inter * hidden, (at + "up_proj").c_str()); + RequireStageSize(layer.down_proj, hidden * inter, (at + "down_proj").c_str()); + } + + vt::Backend& backend = vt::GetBackend(queue.device.type); + Music3DepthDeviceWeights out; + out.pos_embedding = UploadBf16(backend, queue, {&weights.pos_embedding}, + {config.max_position_embeddings, hidden}, release_host, + &out.storage); + out.norm = UploadBf16(backend, queue, {&weights.norm}, {hidden}, release_host, &out.storage); + out.layers.resize(static_cast(config.num_layers)); + for (size_t l = 0; l < out.layers.size(); ++l) { + DepthDecoderLayerWeights& src = weights.layers[l]; + Music3DepthDeviceLayer& dst = out.layers[l]; + dst.input_layernorm = + UploadBf16(backend, queue, {&src.input_layernorm}, {hidden}, release_host, &out.storage); + dst.post_attention_layernorm = UploadBf16(backend, queue, {&src.post_attention_layernorm}, + {hidden}, release_host, &out.storage); + // to_q | to_k | to_v, in THAT order — the order the forward's re-views read + // them back in. Each is [H, H] row-major (out, in), so stacking them is a + // concatenation along the OUTPUT axis and needs no permutation. + dst.qkv = UploadBf16(backend, queue, {&src.to_q, &src.to_k, &src.to_v}, {3 * hidden, hidden}, + release_host, &out.storage); + dst.to_out = + UploadBf16(backend, queue, {&src.to_out}, {hidden, hidden}, release_host, &out.storage); + // gate_proj | up_proj, in THAT order, because `vt::SiluAndMul` computes + // `silu(x[:, :D]) * x[:, D:]` and upstream computes `silu(gate) * up`. + // Swapping these two blocks silently computes `silu(up) * gate`, which is a + // finite and plausible tensor — the gate for it is a mutation, not an + // assertion. + // Held as HOST bf16 bytes: `layers::UnquantizedMlpGateUpMethod` consumes an + // `OwnedTensor` and `ResidentWeight` does the one upload (or aliases, on a + // CPU queue). Same [2I, H] gate-then-up order, same lossless narrowing. + dst.gate_up = PackBf16Owned({&src.gate_proj, &src.up_proj}, {2 * inter, hidden}, + release_host); + dst.down_proj = UploadBf16(backend, queue, {&src.down_proj}, {hidden, inter}, release_host, + &out.storage); + } + return out; +} + +std::vector DepthDecoderAppendDevice(vt::Queue& queue, const DepthDecoderConfig& config, + const Music3DepthDeviceWeights& weights, + const std::vector& inputs_embeds, + int64_t batch, Music3DepthDeviceCache* cache) { + const int64_t hidden = config.hidden_size; + const int64_t inter = config.intermediate_size; + const int64_t heads = config.num_attention_heads; + const int64_t head_dim = config.head_dim(); + // The SAME refusals `DepthDecoderAppend` raises, in the same order and with + // the same words where the words still apply — a caller that swaps arms must + // not have to learn a second vocabulary of errors. + if (cache == nullptr) Fail("MiniMax-Music3: the incremental depth decode needs a cache"); + if (batch <= 0) Fail("MiniMax-Music3: the incremental depth decode needs a positive batch"); + if (static_cast(inputs_embeds.size()) != batch * hidden) { + Fail("MiniMax-Music3: the incremental depth decode got " + + std::to_string(inputs_embeds.size()) + " input values, expected batch*hidden = " + + std::to_string(batch * hidden)); + } + if (!weights.staged()) { + Fail("MiniMax-Music3: the device depth decode was given unstaged weights"); + } + if (static_cast(weights.layers.size()) != config.num_layers) { + Fail("MiniMax-Music3: the depth decoder has " + std::to_string(weights.layers.size()) + + " layer weight sets, expected " + std::to_string(config.num_layers)); + } + if (heads * head_dim != hidden) { + Fail("MiniMax-Music3: hidden_size " + std::to_string(hidden) + + " is not divisible by num_attention_heads " + std::to_string(heads)); + } + + vt::Backend& backend = vt::GetBackend(queue.device.type); + Dev dev{backend, queue}; + if (cache->positions == 0 && cache->keys.empty()) { + // Sized ONCE at `max_position_embeddings` rather than grown per position: + // the ceiling is 16 rows, so the whole history is 16 * H * 2 bytes per + // (layer, row) — 1 MB at the shipped 4-layer batch-2 geometry — and a + // re-allocation mid-sequence would invalidate the views the attention holds. + cache->batch = batch; + cache->hidden = hidden; + cache->layers = config.num_layers; + cache->capacity = config.max_position_embeddings; + const size_t slots = static_cast(config.num_layers * batch); + cache->keys.resize(slots); + cache->values.resize(slots); + for (size_t s = 0; s < 2 * slots; ++s) { + // POOLED, through `DBuf`, and NOT `backend.Alloc` — because this cache is + // a local of `Music3DepthStage` and is therefore built and destroyed once + // per FRAME. At the shipped 4-layer batch-2 geometry a raw pair would be + // 16 `cudaMalloc` plus 16 `cudaFree` per frame, and `cudaFree` + // synchronizes the device; `DBuf`'s own comment says the pool exists for + // exactly that. `ReleaseShared` hands the block a carrier that returns it + // to the pool it came from, which the hand-written deleter this replaced + // could not name. The staged WEIGHTS stay on `backend.Alloc` on purpose: + // they are allocated once and held for the process, and a pool block held + // forever is a pool block withdrawn from the pool. + DBuf block(dev, DType::kBF16, {cache->capacity, hidden}); + const Tensor t = block.t(); + cache->storage.push_back(block.ReleaseShared()); + (s < slots ? cache->keys : cache->values)[s % slots] = t; + } + } + if (cache->batch != batch || cache->hidden != hidden || cache->layers != config.num_layers) { + Fail("MiniMax-Music3: the depth cache holds batch " + std::to_string(cache->batch) + + " / hidden " + std::to_string(cache->hidden) + " / layers " + + std::to_string(cache->layers) + ", asked for " + std::to_string(batch) + " / " + + std::to_string(hidden) + " / " + std::to_string(config.num_layers)); + } + const int64_t position = cache->positions; + if (position >= config.max_position_embeddings) { + Fail("MiniMax-Music3: the depth decoder was given position " + std::to_string(position) + + " but max_position_embeddings is " + std::to_string(config.max_position_embeddings) + + "; pos_embedding has no row for the rest"); + } + const int64_t seq = position + 1; + + // The upload is EXACT: `Store(..., kBFloat16)` already rounded every value the + // host schedule hands across this boundary, so `F32ToBF16` loses nothing. + std::vector staged(inputs_embeds.size()); + for (size_t i = 0; i < inputs_embeds.size(); ++i) staged[i] = vt::F32ToBF16(inputs_embeds[i]); + DBuf hidden_states(dev, DType::kBF16, {batch, hidden}, staged.data()); + + // :138-139 — the SAME learned row `DepthDecoderForward` adds at this position, + // broadcast over the batch by `vt::Add`'s rank-1 form. + const Tensor pos_row = RowView(weights.pos_embedding, position, {hidden}); + vt::Add(queue, hidden_states.t(), hidden_states.t(), pos_row); + + DBuf normed(dev, DType::kBF16, {batch, hidden}); + DBuf qkv(dev, DType::kBF16, {batch, 3 * hidden}); + DBuf attended(dev, DType::kBF16, {batch, hidden}); + DBuf projected(dev, DType::kBF16, {batch, hidden}); + DBuf down(dev, DType::kBF16, {batch, hidden}); + + // #1131's SECOND instrument. Every buffer this call makes resident reports its + // own dtype into one mask, read back by the gate — because the ULP band above + // cannot see a buffer that is too WIDE, and a review proved it by widening + // `normed` to `kF32` and watching all 35 cases stay green. Read the buffers + // rather than restating the constant, or the assertion tests this line instead + // of the allocation. + uint64_t resident = DtypeBit(hidden_states.t().dtype) | DtypeBit(normed.t().dtype) | + DtypeBit(qkv.t().dtype) | DtypeBit(attended.t().dtype) | + DtypeBit(projected.t().dtype) | DtypeBit(down.t().dtype) | + DtypeBit(cache->keys[0].dtype) | DtypeBit(cache->values[0].dtype); + const vt::RmsNormArgs norm_args{1e-6f, /*gemma=*/false}; + const vt::AttentionCrossArgs attn_args{ + static_cast(1.0 / std::sqrt(static_cast(head_dim)))}; + const size_t row_bytes = static_cast(hidden) * sizeof(uint16_t); + + for (int64_t l = 0; l < config.num_layers; ++l) { + const Music3DepthDeviceLayer& layer = weights.layers[static_cast(l)]; + vt::RmsNorm(queue, normed.t(), hidden_states.t(), layer.input_layernorm, norm_args); + // ONE sweep of the merged [3H, H] weight where the host arm sweeps to_q, + // to_k and to_v separately. Row `b` of the result is [q(H) | k(H) | v(H)], + // so each of the three is a CONTIGUOUS H-value slice and needs no copy to be + // viewed — which is what `vt::AttentionCross`'s contiguity requirement asks + // of the query. + vt::MatmulBT(queue, qkv.t(), normed.t(), layer.qkv); + auto* qkv_rows = static_cast(qkv.t().data); + for (int64_t b = 0; b < batch; ++b) { + const size_t slot = static_cast(l * batch + b); + Tensor& kc = cache->keys[slot]; + Tensor& vc = cache->values[slot]; + uint16_t* row = qkv_rows + b * 3 * hidden; + // The K/V of this position, copied into the row's contiguous history. See + // the file header for why the GEMM cannot write here directly, and why an + // 8 KB copy is the right side of that trade. + backend.Copy(queue, static_cast(kc.data) + position * hidden, row + hidden, + row_bytes); + backend.Copy(queue, static_cast(vc.data) + position * hidden, row + 2 * hidden, + row_bytes); + // ONE query row against `seq` cached positions that are ALL at or before + // it, so a causal mask would mask nothing and the non-causal op computes + // upstream's causal result exactly. See the header. + Tensor query = MakeTensor(row, DType::kBF16, queue.device, {1, heads, head_dim}); + Tensor key = MakeTensor(kc.data, DType::kBF16, queue.device, {seq, heads, head_dim}); + Tensor value = MakeTensor(vc.data, DType::kBF16, queue.device, {seq, heads, head_dim}); + Tensor out_row = RowView(attended.t(), b, {1, heads, head_dim}); + vt::AttentionCross(queue, out_row, query, key, value, /*bias=*/nullptr, attn_args); + } + vt::MatmulBT(queue, projected.t(), attended.t(), layer.to_out); + vt::Add(queue, hidden_states.t(), hidden_states.t(), projected.t()); + + vt::RmsNorm(queue, normed.t(), hidden_states.t(), layer.post_attention_layernorm, norm_args); + // ONE sweep of the merged [2I, H] weight through the SHARED merged-GEMM + // seam, laid out gate-then-up so its `vt::SiluAndMul` computes upstream's + // `silu(gate) * up` with no permutation and no half swap. + const layers::UnquantizedMlpGateUpMethod mlp(&layer.gate_up, inter); + DBuf activated = mlp.Apply(dev, normed.t()); + resident |= DtypeBit(activated.t().dtype) | DtypeBit(layer.qkv.dtype) | + DtypeBit(layer.gate_up.dtype) | DtypeBit(layer.down_proj.dtype) | + DtypeBit(layer.to_out.dtype); + vt::MatmulBT(queue, down.t(), activated.t(), layer.down_proj); + vt::Add(queue, hidden_states.t(), hidden_states.t(), down.t()); + } + + // :142 + DBuf normed_out(dev, DType::kBF16, {batch, hidden}); + vt::RmsNorm(queue, normed_out.t(), hidden_states.t(), weights.norm, norm_args); + resident |= DtypeBit(normed_out.t().dtype) | DtypeBit(weights.norm.dtype) | + DtypeBit(weights.pos_embedding.dtype); + std::vector host(static_cast(batch * hidden)); + normed_out.Download(dev, host.data()); + + cache->positions = seq; + // OR-ed ONCE, at the end, so a call that threw part way through never reports + // a partial residency as if it were the whole forward's. + g_resident_dtypes.fetch_or(resident, std::memory_order_relaxed); + g_forward_count.fetch_add(1, std::memory_order_relaxed); + std::vector out(host.size()); + for (size_t i = 0; i < host.size(); ++i) out[i] = vt::BF16ToF32(host[i]); + return out; +} + +} // namespace music3 +} // namespace models +} // namespace vllm diff --git a/src/vllm/model_executor/models/minimax_music3_llm.cpp b/src/vllm/model_executor/models/minimax_music3_llm.cpp index 4f72476c3..cafc92620 100644 --- a/src/vllm/model_executor/models/minimax_music3_llm.cpp +++ b/src/vllm/model_executor/models/minimax_music3_llm.cpp @@ -416,7 +416,8 @@ std::vector Music3DepthStage(const std::vector& last_hidden_condit const std::vector& last_hidden_unconditional, int64_t frame_index, const Music3ArWeights& weights, const Music3CodeSampler& sampler, - std::vector* out_frame_codes) { + std::vector* out_frame_codes, + const Music3DepthDeviceArm& device_arm) { // `_generate_depth_codes` (encoders.py:117-142). The two prompt rows share // every sequence entry EXCEPT position 0, which is each row's own // `projection(last_hidden)` — that is the whole of what the unconditional @@ -431,6 +432,15 @@ std::vector Music3DepthStage(const std::vector& last_hidden_condit if (out_frame_codes->size() != 1) { Fail("MiniMax-Music3: the depth stage starts from the frame's semantic code alone"); } + // Half an arm is a caller that thinks it asked for the device and did not. + // Refused by name rather than silently falling back to the host loop, which + // would be a 4.4x slowdown wearing a correct answer. + if (device_arm.half_set()) { + Fail("MiniMax-Music3: the depth stage's device arm needs BOTH a queue and staged weights; " + "got " + + std::string(device_arm.queue != nullptr ? "a queue and no weights" + : "weights and no queue")); + } profile::Timer depth_timer("ar.depth_stage", /*span=*/true); const int32_t semantic = (*out_frame_codes)[0]; const std::vector semantic_embed = @@ -452,6 +462,18 @@ std::vector Music3DepthStage(const std::vector& last_hidden_condit // The two CFG rows go through as ONE batch-2 call — which is upstream's own // shape, `(2, 2..8, 4096)` — so every weight sweep serves both branches. DepthDecoderCache cache; + Music3DepthDeviceCache device_cache; + // THE PRODUCTION SELECTION (#1309, spec §19.5). Deleting this lambda's device + // branch is the reachability mutation, and the composed-stage gate goes RED on + // it — which is the leg #1131 records as missing for the DiT arm. + const auto append = [&](const std::vector& embeds) { + if (device_arm.engaged()) { + return DepthDecoderAppendDevice(*device_arm.queue, config, *device_arm.depth, embeds, + /*batch=*/2, &device_cache); + } + return DepthDecoderAppend(embeds, /*batch=*/2, config, weights.depth, ArCompute::kBFloat16, + &cache); + }; // The depth sequence's first two rows: each branch's own // `projection(last_hidden)` at position 0, and the SHARED @@ -476,8 +498,7 @@ std::vector Music3DepthStage(const std::vector& last_hidden_condit // row of a length-2 sequence first, so position 0's own output is never used. { profile::Timer forward_timer("ar.depth_forward"); - DepthDecoderAppend(std::vector(prefix.begin(), prefix.begin() + 2 * H), /*batch=*/2, - config, weights.depth, ArCompute::kBFloat16, &cache); + append(std::vector(prefix.begin(), prefix.begin() + 2 * H)); } std::vector next(static_cast(2 * H)); @@ -488,8 +509,7 @@ std::vector Music3DepthStage(const std::vector& last_hidden_condit std::vector states; { profile::Timer forward_timer("ar.depth_forward"); - states = DepthDecoderAppend(next, /*batch=*/2, config, weights.depth, - ArCompute::kBFloat16, &cache); + states = append(next); } std::vector hidden_rows[2]; hidden_rows[0].assign(states.begin(), states.begin() + H); @@ -542,6 +562,30 @@ std::vector Music3DepthStage(const std::vector& last_hidden_condit return depth_hidden; } +// --------------------------------------------------------------------------- +// The depth decoder's device arm, selected +// --------------------------------------------------------------------------- + +Music3DepthDeviceArm Music3SelectDepthArm(vt::Queue& queue, const DepthDecoderConfig& config, + DepthDecoderWeights& weights, bool release_host, + Music3DepthDeviceWeights* staged) { + // See minimax_music3_llm.h for why this is a function and not an `if` in the + // engine: the engine's condition is false on every runner CI owns, and #1131 + // is what an ungateable selection costs. + if (staged == nullptr) { + Fail("MiniMax-Music3: the depth arm selection needs somewhere to stage into"); + } + Music3DepthDeviceArm arm; + if (queue.device.type == vt::DeviceType::kCPU) return arm; + // Timed HERE rather than at the call site so the span exists only when + // something is actually staged; a CPU queue returns above and records nothing. + profile::Timer stage_timer("ar.depth_staging"); + *staged = StageMusic3DepthWeights(queue, config, weights, release_host); + arm.queue = &queue; + arm.depth = staged; + return arm; +} + // --------------------------------------------------------------------------- // The loop // --------------------------------------------------------------------------- @@ -550,7 +594,8 @@ Music3ArResult Music3GenerateFrameHiddens(const std::vector& prompt_ids int64_t max_frames, const Music3ArWeights& weights, const Music3CodeSampler& sampler, - vt::Queue& queue) { + vt::Queue& queue, + const Music3DepthDeviceArm& device_arm) { if (!sampler) Fail("MiniMax-Music3: the autoregressive loop needs a code sampler"); if (max_frames <= 0) { Fail("MiniMax-Music3: the autoregressive loop needs a positive frame budget, got " + @@ -634,7 +679,7 @@ Music3ArResult Music3GenerateFrameHiddens(const std::vector& prompt_ids const std::vector depth_hidden = Music3DepthStage( std::vector(hidden.begin(), hidden.begin() + static_cast(H)), std::vector(hidden.begin() + static_cast(H), hidden.end()), - frame_index, weights, sampler, &frame_codes); + frame_index, weights, sampler, &frame_codes, device_arm); result.codes.insert(result.codes.end(), frame_codes.begin(), frame_codes.end()); ++result.calls; diff --git a/src/vllm/model_executor/models/minimax_music3_speech.cpp b/src/vllm/model_executor/models/minimax_music3_speech.cpp index 6f10f0d04..6e0fd6494 100644 --- a/src/vllm/model_executor/models/minimax_music3_speech.cpp +++ b/src/vllm/model_executor/models/minimax_music3_speech.cpp @@ -578,20 +578,46 @@ class Music3SpeechEngine final : public multimodal::SpeechEngine { // `queue_` and the ONLY thing the device selector changes. // // WHAT MOVES: the 8.6B `Qwen3ForCausalLM` half, through the shared - // `Qwen3DenseModel::ForwardEmbeds` five registrations already ride. WHAT - // DOES NOT: the 0.65B RVQ depth decoder and the whole acoustic half, - // which are host reference loops — see minimax_music3_ar.h and - // vocoder1d.h for exactly which pieces are owed and why. + // `Qwen3DenseModel::ForwardEmbeds` five registrations already ride, AND — + // since #1309 — the 0.65B RVQ depth decoder, which was 48.4 % of a run + // (spec §19.1). WHAT DOES NOT: the depth decoder's projection, audio + // heads and feedback embedding, ~1.6 % of that stage and owed by §19.7; + // and the whole acoustic half's host reference loops — see + // minimax_music3_ar.h and vocoder1d.h for which pieces are owed and why. + // + // NON-const, because the depth arm below STAGES OUT OF IT. const auto load_t0 = profile::Now(); - const Music3ArWeights ar = Music3LoadArWeights(paths_, config_); + Music3ArWeights ar = Music3LoadArWeights(paths_, config_); profile::AddSince("load.ar_weights", load_t0); profile::Mark("ar.weights_loaded"); const std::vector prompt_ids = ar.Encode(request.prompt); + + // THE PRODUCTION SELECTION for the depth decoder, on the SAME switch the + // DiT arm rides: `--speech-device 1` resolves `queue_` to the platform's + // device, and a non-CPU queue takes the device arm. There is no separate + // flag and no environment variable, because a capability behind an option + // nothing turns on is the shape `.agents/reachability.md` calls dead. + // + // The rule itself lives in `Music3SelectDepthArm` rather than in an `if` + // here, and that placement is the #1131 repair: the condition `queue_` has + // to satisfy is false on every runner CI owns, so a branch written at this + // line is unreachable from any gate, while the function is driven by + // `test_minimax_music3_ar` on both sides of it. The DiT block below still + // carries the untestable shape and #1131 still owns it. + // + // `release_host` is TRUE: the staged tensors are the ONLY thing the host + // append loop reads, and it is not called when the arm is engaged. The + // projection, the audio embeddings and the audio heads — which this stage + // still reads on the host — are not staged and are not released. + Music3DepthDeviceWeights staged_depth; + const Music3DepthDeviceArm depth_arm = Music3SelectDepthArm( + queue_, ar.depth_config, ar.depth, /*release_host=*/true, &staged_depth); Music3ArResult generated; { profile::Timer ar_timer("ar.TOTAL_loop", /*span=*/true); - generated = Music3GenerateFrameHiddens( - prompt_ids, request.max_frames, ar, Music3SeededSampler(request.seed), queue_); + generated = Music3GenerateFrameHiddens(prompt_ids, request.max_frames, ar, + Music3SeededSampler(request.seed), queue_, + depth_arm); } profile::Mark("ar.loop_done"); frame_hiddens = std::move(generated.frame_hiddens); diff --git a/tests/vllm/models/test_minimax_music3_ar.cpp b/tests/vllm/models/test_minimax_music3_ar.cpp index ad460ed62..492b41675 100644 --- a/tests/vllm/models/test_minimax_music3_ar.cpp +++ b/tests/vllm/models/test_minimax_music3_ar.cpp @@ -23,6 +23,7 @@ // attention 4 / 65, and a frame row that keeps depth position 0 reds 1 / 24. #include +#include #include #include #include @@ -1173,6 +1174,672 @@ TEST_CASE("music3 ar: the COMPOSED depth stage is BIT-IDENTICAL to the schedule << " reference values non-zero, " << calls_new << " draws"); } +// --------------------------------------------------------------------------- +// The DEVICE arm (#1309, spec §19) +// --------------------------------------------------------------------------- +// +// These run on a `vt::Queue` whose device is kCPU. That is not a compromise, it +// is the point: every op the device forward composes — kMatmulBT, kRmsNorm, +// kAttentionCross, kSiluAndMul, kAdd — carries a CPU provider AND a CUDA one, so +// the composition, the merged weight layouts, the cache indexing and the +// PRODUCTION SELECTION are all gated here with no GPU and no checkpoint. +// +// #1131 is why the selection is gated rather than only the forward. The DiT +// device arm's kernels are gated and its switch is not, so `on_device = false` +// leaves every suite green — the two arms agree numerically BY DESIGN, and no +// gate there ever asked WHICH ONE RAN. `Music3DepthDeviceForwardCount()` is what +// asks it here. +// +// What these cases do NOT reach is the CUDA kernels themselves, and that is said +// rather than implied: the CUDA `AttentionCross` uses an online-softmax +// recurrence where the CPU one uses three passes, and cuBLASLt splits K by an +// algorithm no CPU provider has. Spec §19.6 owes that leg to `thor:gpu0`. + +// The spacing of bf16 at `value`'s magnitude. bf16 keeps 8 significand bits +// (1 implicit + 7 stored), so a value in [2^k, 2^(k+1)) has ULP 2^(k-7). +double Bf16Ulp(double value) { + if (value == 0.0 || !std::isfinite(value)) return std::ldexp(1.0, -133); + int exponent = 0; + std::frexp(std::abs(value), &exponent); // |v| = m * 2^e, m in [0.5, 1) + return std::ldexp(1.0, exponent - 8); +} + +// A CPU queue. `vt::Queue`'s CPU form carries a null handle by construction — +// the speech engine builds exactly this when `--speech-device 0`. +vt::Queue CpuQueue() { return vt::Queue{vt::Device{vt::DeviceType::kCPU, 0}, nullptr}; } + +// A deterministic reduced geometry, WIDER than the goldens on both axes that a +// merged weight layout could get wrong: 8 heads of 8 (so `heads * head_dim` is +// not `head_dim`), and an intermediate size that is not a multiple of the hidden +// size (so a gate/up half swap cannot be masked by a symmetric shape). +m3::DepthDecoderConfig DeviceArmConfig() { + m3::DepthDecoderConfig config; + config.hidden_size = 64; + config.num_layers = 3; + config.num_attention_heads = 8; + config.intermediate_size = 96; + config.audio_vocab_size = 32; + config.num_codebooks = 8; + config.max_position_embeddings = 8; + return config; +} + +// Weights whose values are ALREADY bf16-exact, because that is what the loader +// hands the arms: `AtRuntimeDtype` rounds every AR-half tensor through bf16 into +// its f32 carrier. Drawing f32 noise here instead would make the device arm's +// staging lossy for a reason the shipped path does not have, and would price a +// rounding this row does not perform. +std::vector DeviceArmFill(size_t n, uint32_t* state, float scale) { + std::vector v(n); + for (size_t i = 0; i < n; ++i) { + *state = *state * 1664525u + 1013904223u; + const float raw = + static_cast(static_cast(*state >> 8) / 8388608.0 - 1.0) * scale; + v[i] = vt::BF16ToF32(vt::F32ToBF16(raw)); + } + return v; +} + +m3::DepthDecoderWeights DeviceArmWeights(const m3::DepthDecoderConfig& config, uint32_t* state) { + const size_t hidden = static_cast(config.hidden_size); + const size_t inter = static_cast(config.intermediate_size); + m3::DepthDecoderWeights weights; + weights.audio_embeddings = DeviceArmFill( + static_cast(config.audio_vocab_size * config.residual_codebooks()) * hidden, state, + 0.5f); + weights.projection = DeviceArmFill(hidden * hidden, state, 0.25f); + weights.pos_embedding = + DeviceArmFill(static_cast(config.max_position_embeddings) * hidden, state, 0.5f); + weights.norm = DeviceArmFill(hidden, state, 0.5f); + for (int64_t l = 0; l < config.num_layers; ++l) { + m3::DepthDecoderLayerWeights layer; + layer.input_layernorm = DeviceArmFill(hidden, state, 0.5f); + layer.post_attention_layernorm = DeviceArmFill(hidden, state, 0.5f); + layer.to_q = DeviceArmFill(hidden * hidden, state, 0.25f); + layer.to_k = DeviceArmFill(hidden * hidden, state, 0.25f); + layer.to_v = DeviceArmFill(hidden * hidden, state, 0.25f); + layer.to_out = DeviceArmFill(hidden * hidden, state, 0.25f); + layer.gate_proj = DeviceArmFill(inter * hidden, state, 0.25f); + layer.up_proj = DeviceArmFill(inter * hidden, state, 0.25f); + layer.down_proj = DeviceArmFill(hidden * inter, state, 0.25f); + weights.layers.push_back(std::move(layer)); + } + for (int64_t h = 0; h < config.residual_codebooks(); ++h) { + weights.audio_heads.push_back( + DeviceArmFill(static_cast(config.audio_vocab_size) * hidden, state, 0.25f)); + } + return weights; +} + +// The seeds this gate measures over, and WHY there is more than one. The bound +// below was originally fitted to the first of them, and a review falsified it by +// changing NOTHING BUT THE SEED: same distribution, same geometry, no defect, and +// the shipped `mean <= 4.0` red on three of six equally valid draws. A tolerance +// that a redraw of the reference weights can fail is measuring the draw, not the +// arm. §19.4b carries the per-seed table and the arithmetic behind the bounds. +constexpr uint32_t kDeviceArmSeeds[] = {0x9E3779B9u, 0x2468ACE0u, 0x00000001u, + 0x13579BDFu, 0xDEADBEEFu, 0x51ED2701u}; +constexpr size_t kNumDeviceArmSeeds = sizeof(kDeviceArmSeeds) / sizeof(kDeviceArmSeeds[0]); + +// One draw's worth of device-vs-host deviation. +// +// TWO BUCKETS, AND THEY SUM. A reference value of EXACTLY zero has no ULP — +// bf16's spacing at zero is the denormal floor, so `|got| / Bf16Ulp(0)` reads +// ~1e35 for an absolute difference of 1e-5 and the statistic stops meaning +// anything about the arm. Seed `0x51ED2701` draws one such value in 1024 and the +// CORRECT arm reads mean 8.8e32 on it. So a zero reference is measured +// ABSOLUTELY instead, and the two counts are asserted to sum to the compared +// total: a defect that produced zeros would have to grow that bucket to hide in +// it, and growing it is a failure. +struct DeviceArmBand { + double worst = 0.0; // bf16 ULP, over the non-zero references only + double mean = 0.0; // bf16 ULP, over the non-zero references only + double median = 0.0; // bf16 ULP, over the non-zero references only + double worst_at_zero = 0.0; // absolute, over the zero references only + size_t compared = 0; + size_t nonzero = 0; + size_t at_zero = 0; + uint64_t forwards = 0; +}; + +// Run BOTH arms over the whole position schedule at one seed and measure the gap. +DeviceArmBand MeasureDeviceArmBand(const m3::DepthDecoderConfig& config, uint32_t seed) { + uint32_t state = seed; + m3::DepthDecoderWeights host_weights = DeviceArmWeights(config, &state); + m3::DepthDecoderWeights stage_source = host_weights; + + vt::Queue queue = CpuQueue(); + // `release_host=false`: this gate compares the two arms, so the host copy must + // survive. A serving path passes true. + const m3::Music3DepthDeviceWeights staged = + m3::StageMusic3DepthWeights(queue, config, stage_source, /*release_host=*/false); + REQUIRE(staged.staged()); + + const int64_t batch = 2; + const int64_t H = config.hidden_size; + m3::DepthDecoderCache host_cache; + m3::Music3DepthDeviceCache device_cache; + DeviceArmBand band; + std::vector ulps; + ulps.reserve(static_cast(config.max_position_embeddings * batch * H)); + const uint64_t before = m3::Music3DepthDeviceForwardCount(); + + for (int64_t position = 0; position < config.max_position_embeddings; ++position) { + // The two CFG rows differ, which is what makes a schedule that served one to + // both detectable at all. + const std::vector embeds = DeviceArmFill(static_cast(batch * H), &state, 1.0f); + const std::vector want = m3::DepthDecoderAppend( + embeds, batch, config, host_weights, m3::ArCompute::kBFloat16, &host_cache); + const std::vector got = + m3::DepthDecoderAppendDevice(queue, config, staged, embeds, batch, &device_cache); + REQUIRE(got.size() == want.size()); + for (size_t i = 0; i < want.size(); ++i) { + const double absolute = std::abs(static_cast(got[i]) - want[i]); + ++band.compared; + if (want[i] == 0.0f) { + band.worst_at_zero = std::max(band.worst_at_zero, absolute); + ++band.at_zero; + continue; + } + const double ulp = absolute / Bf16Ulp(want[i]); + band.worst = std::max(band.worst, ulp); + band.mean += ulp; + ulps.push_back(ulp); + ++band.nonzero; + } + } + band.forwards = m3::Music3DepthDeviceForwardCount() - before; + band.mean /= static_cast(band.nonzero == 0 ? 1 : band.nonzero); + if (!ulps.empty()) { + const size_t mid = ulps.size() / 2; + std::nth_element(ulps.begin(), ulps.begin() + static_cast(mid), ulps.end()); + band.median = ulps[mid]; + } + return band; +} + +// THE BOUNDS, placed from a MULTI-SEED measurement against a mutation battery +// rather than from the deviation one draw happened to produce. §19.4b carries +// the whole table; the two things that decide the numbers are these. +// +// THE CORRECT ARM, over six seeds: +// +// seed worst mean median note +// 0x9E3779B9 110 2.095 1 the seed the old bound was fitted to +// 0x2468ACE0 435 3.804 1 +// 0x00000001 1110 5.755 1 RED under the old `mean <= 4.0` +// 0x13579BDF 3663 7.154 1 RED under the old `mean <= 4.0` +// 0xDEADBEEF 7340 9.904 1 RED under the old `mean <= 4.0` +// 0x51ED2701 939 3.103 1 one reference is exactly zero +// +// FIVE STRUCTURAL MUTATIONS, each over the same six seeds, worst case first: +// +// mutation mean range median range +// wrong attention scale 19.6 - 71.3 4 - 4 +// gate/up half swap 52.5 - 158.8 12 - 15 +// K/V cache row collision 166.4 - 376.1 3 - 4 +// dropped position embedding 357.8 - 1487 76 - 90 +// q|k|v merge order swapped 400.0 - 1731 104 - 131 +// +// WHY THE MEDIAN IS THE PRIMARY GATE. It is EXACTLY 1 bf16 ULP at all six seeds +// for the correct arm — a constant, not a distribution — because the deviation +// this arm actually carries is one rounding-polarity tick per element, and the +// seed only changes the tail. Every defect is at least 3. So the median's window +// is (1, 3] and it cannot be moved by a redraw, which is precisely the property +// the old single-seed bound lacked. +// +// WHY THE MEAN IS STILL GATED, and what it costs. The mean is tail-sensitive, so +// it catches a SPARSE defect that leaves the middle of the distribution alone and +// the median would not see. Its window is narrow and asymmetric: the correct +// arm's worst draw is 9.904 and the tightest defect is 19.6 — the wrong attention +// scale at `0xDEADBEEF`, which is the same seed that produces the correct arm's +// own worst mean. 15 sits inside that window, 1.51x above every correct draw +// measured and 1.31x below every defect draw measured. That is a real margin and +// it is not a large one, and a seventh seed drawing a correct mean above 15 is +// possible in a way that one drawing a median above 2 is not. +// +// WHY `worst` IS NOT GATED AT ALL. It cannot discriminate. A CORRECT arm reads +// worst 7340 at `0xDEADBEEF` while the gate/up half swap reads 6641 and the wrong +// attention scale 6865 at the seed each is worst-separated on, so any `worst` +// bound loose enough to admit a correct implementation admits two structural +// defects. It is reported, and it carries only a canary far above any draw here, +// because a value that has gone non-finite must not read as rounding. +constexpr double kDeviceArmMedianUlpBound = 2.0; +constexpr double kDeviceArmMeanUlpBound = 15.0; +constexpr double kDeviceArmWorstUlpCanary = 1.0e6; +// The zero-reference bucket, measured ABSOLUTELY because it has no ULP. See +// `DeviceArmBand`. +constexpr double kDeviceArmZeroAbsBound = 1.0e-2; + +TEST_CASE("music3 ar: the DEVICE depth decode tracks the host arm inside a bf16 band") { + // NOT bitwise, and the spec says so BEFORE this code rather than after a + // surprise (§19.4). Three reasons, each sufficient alone: the host keeps a + // sequential `double` per output element where `vt::MatmulBT` accumulates in + // f32; the reduction re-associates; and `vt::RmsNorm` keeps full f32 precision + // across the weight multiply where our `RmsNorm` mirrors upstream's TWO + // roundings. The first of those makes the device arm the CLOSER mirror of + // torch, which accumulates bf16 matmuls in f32. + // + // So the bound is in bf16 ULPs OF THE REFERENCE VALUE, not in absolute units: + // four layers of activations span orders of magnitude and an absolute bound + // would be vacuous at the top and impossible at the bottom. + // + // And it is asserted over SIX DRAWS, because the single-draw version of this + // case passed while a correct arm failed on three other seeds (§19.4b). + const m3::DepthDecoderConfig config = DeviceArmConfig(); + double worst_over_seeds = 0.0; + double worst_mean = 0.0; + double worst_median = 0.0; + size_t total_compared = 0; + size_t total_nonzero = 0; + size_t total_at_zero = 0; + double worst_at_zero = 0.0; + uint64_t total_forwards = 0; + + for (size_t s = 0; s < kNumDeviceArmSeeds; ++s) { + const uint32_t seed = kDeviceArmSeeds[s]; + CAPTURE(seed); + const DeviceArmBand band = MeasureDeviceArmBand(config, seed); + worst_over_seeds = std::max(worst_over_seeds, band.worst); + worst_mean = std::max(worst_mean, band.mean); + worst_median = std::max(worst_median, band.median); + total_compared += band.compared; + total_nonzero += band.nonzero; + total_at_zero += band.at_zero; + worst_at_zero = std::max(worst_at_zero, band.worst_at_zero); + total_forwards += band.forwards; + + CHECK_MESSAGE(band.median <= kDeviceArmMedianUlpBound, + "seed " << seed << ": device/host MEDIAN deviation " << band.median + << " bf16 ULP exceeds " << kDeviceArmMedianUlpBound + << "; the correct arm is exactly 1 at every seed measured, so this " + "is a STRUCTURAL defect and not a redraw"); + CHECK_MESSAGE(band.mean <= kDeviceArmMeanUlpBound, + "seed " << seed << ": device/host MEAN deviation " << band.mean + << " bf16 ULP exceeds " << kDeviceArmMeanUlpBound + << ", which is a STRUCTURAL defect, not rounding"); + CHECK_MESSAGE(band.worst <= kDeviceArmWorstUlpCanary, + "seed " << seed << ": device/host WORST deviation " << band.worst + << " bf16 ULP is past the non-finite canary"); + CHECK_MESSAGE(band.worst_at_zero <= kDeviceArmZeroAbsBound, + "seed " << seed << ": where the reference is EXACTLY zero the device arm is " + << band.worst_at_zero << " away, past " << kDeviceArmZeroAbsBound); + // Per draw, not only in aggregate, and the buckets SUM. An all-zero + // reference satisfies any tolerance; so does a forward that never ran; and + // so does a defect that quietly moved every element into the un-gated + // zero-reference bucket. + CHECK_MESSAGE(band.nonzero > 0, "seed " << seed << ": the reference block is all zeros"); + const size_t bucketed = band.nonzero + band.at_zero; + CHECK_MESSAGE(bucketed == band.compared, + "seed " << seed << ": " << band.nonzero << " + " << band.at_zero + << " buckets do not sum to " << band.compared << " compared values"); + CHECK_MESSAGE(band.compared == + static_cast(config.max_position_embeddings * 2 * config.hidden_size), + "seed " << seed << ": the position sweep did not compare every position"); + CHECK_MESSAGE(band.forwards == static_cast(config.max_position_embeddings), + "seed " << seed << ": the DEVICE forward ran " << band.forwards + << " times, expected " << config.max_position_embeddings); + MESSAGE("seed " << seed << ": worst " << band.worst << " bf16 ULP, mean " << band.mean + << ", median " << band.median << ", " << band.nonzero << "/" << band.compared + << " references non-zero, " << band.at_zero << " at zero (worst absolute " + << band.worst_at_zero << ")"); + } + + // THE SWEEP ITSELF HAS TEETH. A loop that silently measured one seed six times, + // or zero seeds, would satisfy every bound above. + CHECK_MESSAGE(kNumDeviceArmSeeds >= 6, + "a multi-seed bound needs more than one seed, got " << kNumDeviceArmSeeds); + const uint64_t want_forwards = + static_cast(config.max_position_embeddings) * kNumDeviceArmSeeds; + CHECK(total_forwards == want_forwards); + const size_t total_bucketed = total_nonzero + total_at_zero; + CHECK(total_bucketed == total_compared); + // The un-gated bucket must stay NEGLIGIBLE. It is the one place a defect could + // sit without being measured in ULPs, so its size is a gate of its own. + CHECK_MESSAGE(total_at_zero * 100 < total_compared, + total_at_zero << " of " << total_compared + << " references are exactly zero, so the ULP statistic is being " + "computed over a shrinking minority"); + MESSAGE("device depth decode over " << kNumDeviceArmSeeds << " seeds: " << total_compared + << " values (" << total_at_zero << " at zero, worst " + << worst_at_zero << " absolute), worst " + << worst_over_seeds + << " bf16 ULP (reported, not gated), worst mean " + << worst_mean << " against " << kDeviceArmMeanUlpBound + << ", worst median " << worst_median << " against " + << kDeviceArmMedianUlpBound); +} + +TEST_CASE("music3 ar: the device depth arm is bf16 RESIDENT, on the weights AND the activations") { + // #1131 offers two instruments — "invocation count OR resident dtype" — and + // this row took only the first until a review took the second one's absence + // and proved it mattered: widening a single activation buffer to `kF32` left + // the ULP band, the drawn codes and every case in this file GREEN while the + // path moved twice the bytes. That is AGENTS.md's "a token gate cannot detect + // a dtype that is too WIDE" landing on the very row whose thesis is a dtype, + // and it is §19.2a's finding about the HOST arm arriving inside the arm that + // exists to fix it. + const m3::DepthDecoderConfig config = DeviceArmConfig(); + uint32_t state = 0x5EEDD7EEu; + m3::DepthDecoderWeights source = DeviceArmWeights(config, &state); + vt::Queue queue = CpuQueue(); + const m3::Music3DepthDeviceWeights staged = + m3::StageMusic3DepthWeights(queue, config, source, /*release_host=*/false); + REQUIRE(staged.staged()); + + // THE STAGED WEIGHTS, read off the tensors rather than inferred from the + // numbers. f32 weights would be MORE precise and would sail through every + // tolerance in this file. + CHECK(staged.pos_embedding.dtype == vt::DType::kBF16); + CHECK(staged.norm.dtype == vt::DType::kBF16); + REQUIRE(static_cast(staged.layers.size()) == config.num_layers); + for (const m3::Music3DepthDeviceLayer& layer : staged.layers) { + CHECK(layer.input_layernorm.dtype == vt::DType::kBF16); + CHECK(layer.post_attention_layernorm.dtype == vt::DType::kBF16); + CHECK(layer.qkv.dtype == vt::DType::kBF16); + CHECK(layer.to_out.dtype == vt::DType::kBF16); + CHECK(layer.gate_up.dtype == vt::DType::kBF16); + CHECK(layer.down_proj.dtype == vt::DType::kBF16); + } + + // THE ACTIVATIONS AND THE K/V CACHE, which no caller can see from outside, so + // the forward reports what it allocated. One call is enough to populate the + // mask; the mask is monotone over the process, so a widened buffer anywhere in + // this binary lands here. + const int64_t H = config.hidden_size; + const std::vector embeds = DeviceArmFill(static_cast(2 * H), &state, 1.0f); + m3::Music3DepthDeviceCache cache; + const std::vector out = + m3::DepthDecoderAppendDevice(queue, config, staged, embeds, 2, &cache); + CHECK(out.size() == static_cast(2 * H)); + + const uint64_t resident = m3::Music3DepthDeviceResidentDtypes(); + const uint64_t bf16_only = uint64_t{1} << static_cast(vt::DType::kBF16); + CHECK_MESSAGE(resident != 0, + "the resident-dtype instrument reported NOTHING, so it measured nothing"); + CHECK_MESSAGE(resident == bf16_only, + "the device depth forward made buffers resident in dtypes other than bf16: " + "mask 0x" + << std::hex << resident << std::dec << ", expected 0x" << std::hex + << bf16_only << std::dec + << " — a WIDER buffer is invisible to every tolerance in this file"); +} + +TEST_CASE("music3 ar: the depth arm SELECTION stages on a device queue and never on a CPU one") { + // THE #1131 REPAIR. The engine used to decide this with an `if` on + // `queue_.device.type`, and that condition is false on every runner CI owns — + // so the branch was the one line no gate could enter, which is #1131's shape + // reproduced by the change that cites #1131 as its reason for existing. The + // rule now lives in `Music3SelectDepthArm`, which runs on BOTH sides of that + // condition and is therefore drivable here. + const m3::DepthDecoderConfig config = DeviceArmConfig(); + uint32_t state = 0x0BADC0DEu; + const m3::DepthDecoderWeights depth = DeviceArmWeights(config, &state); + + SUBCASE("a CPU queue stages NOTHING and keeps the host reference arm") { + vt::Queue queue = CpuQueue(); + m3::DepthDecoderWeights source = depth; + m3::Music3DepthDeviceWeights staged; + const uint64_t before = m3::Music3DepthDeviceForwardCount(); + const m3::Music3DepthDeviceArm arm = + m3::Music3SelectDepthArm(queue, config, source, /*release_host=*/true, &staged); + CHECK_FALSE(arm.engaged()); + CHECK_FALSE(arm.half_set()); + CHECK_FALSE(staged.staged()); + CHECK(m3::Music3DepthDeviceForwardCount() == before); + // `release_host` was TRUE and NOTHING may have been released, because the + // host loop is what this queue selected and the host loop reads these. + REQUIRE(source.layers.size() == depth.layers.size()); + CHECK(source.pos_embedding.size() == depth.pos_embedding.size()); + CHECK(source.layers[0].to_q.size() == depth.layers[0].to_q.size()); + CHECK(source.layers[0].gate_proj.size() == depth.layers[0].gate_proj.size()); + } + + SUBCASE("EVERY non-CPU device stages or refuses, and never silently falls back") { + // THE THIRD OUTCOME IS THE DEFECT. A device queue must come back with an + // ENGAGED arm, or the staging must REFUSE by name because this build has no + // provider for that device. What must not happen is a quiet return of a + // disengaged arm, because that is a caller that asked for the device, was + // given the host loop, and was told nothing — #1131 exactly. + // + // `kCUDA` is deliberately NOT in this list. On a CUDA build without a device + // the staging would fail INSIDE the CUDA runtime, and a CUDA call designed + // to fail latches a sticky error that the next unrelated kernel reports as + // its own. Every entry below takes the identical branch, so the rule is + // covered and the latch is not armed. + constexpr vt::DeviceType kNonCpuDevices[] = { + vt::DeviceType::kMETAL, vt::DeviceType::kVULKAN, vt::DeviceType::kXPU, + vt::DeviceType::kROCM, vt::DeviceType::kTENSTORRENT}; + int engaged_count = 0; + int refused_count = 0; + for (vt::DeviceType type : kNonCpuDevices) { + CAPTURE(vt::DeviceTypeName(type)); + vt::Queue queue{vt::Device{type, 0}, nullptr}; + m3::DepthDecoderWeights source = depth; + m3::Music3DepthDeviceWeights staged; + bool engaged = false; + bool refused = false; + try { + const m3::Music3DepthDeviceArm arm = + m3::Music3SelectDepthArm(queue, config, source, /*release_host=*/false, &staged); + engaged = arm.engaged(); + } catch (const std::runtime_error&) { + refused = true; + } + engaged_count += engaged ? 1 : 0; + refused_count += refused ? 1 : 0; + // Named, because doctest cannot decompose a `||` inside a CHECK. + const bool selection_fired = engaged || refused; + CHECK_MESSAGE(selection_fired, "device '" + << vt::DeviceTypeName(type) + << "' quietly took the HOST arm: the selection did " + "not fire and nothing said so"); + } + // The loop must have RUN. A list that emptied, or a body that threw before + // the first CHECK, would leave every assertion above unexecuted. + const int outcomes = engaged_count + refused_count; + CHECK(outcomes == static_cast(sizeof(kNonCpuDevices) / sizeof(kNonCpuDevices[0]))); + MESSAGE("non-CPU selection: " << engaged_count << " staged, " << refused_count + << " refused by name"); + } + + SUBCASE("a null staging slot is refused rather than dereferenced") { + vt::Queue queue{vt::Device{vt::DeviceType::kXPU, 0}, nullptr}; + m3::DepthDecoderWeights source = depth; + CHECK_THROWS_AS( + m3::Music3SelectDepthArm(queue, config, source, /*release_host=*/false, nullptr), + std::runtime_error); + } +} + +TEST_CASE("music3 ar: the COMPOSED depth stage TAKES the device arm and draws the same codes") { + // The reachability gate. It drives the PRODUCTION `Music3DepthStage` — the one + // `Music3GenerateFrameHiddens` calls, which is how the registered speech family + // reaches this row — with the device arm ENGAGED, and it asks the question + // #1131 says nobody asked of the DiT arm: which arm ran? + // + // A numeric comparison alone cannot answer that, because the two arms agree by + // design. So the counter is checked, and it is checked for an EXACT value + // rather than for movement: `num_codebooks` calls a frame, which is §16.2's 8. + const m3::DepthDecoderConfig config = DeviceArmConfig(); + uint32_t state = 0x51ED2701u; + m3::DepthDecoderWeights depth = DeviceArmWeights(config, &state); + const int64_t H = config.hidden_size; + const size_t hidden = static_cast(H); + + m3::Music3ArWeights weights; + weights.depth_config = config; + weights.depth = depth; + weights.lm_config.hidden_size = H; + const int64_t vocab = m3::kAudioCodeOffset + 64; + weights.lm_config.vocab_size = vocab; + { + std::vector bytes(static_cast(vocab) * hidden * sizeof(uint16_t), 0); + uint16_t* const rows = reinterpret_cast(bytes.data()); + const std::vector table = + DeviceArmFill(static_cast(vocab - m3::kAudioCodeOffset) * hidden, &state, 0.5f); + for (int64_t t = m3::kAudioCodeOffset; t < vocab; ++t) { + for (int64_t j = 0; j < H; ++j) { + rows[t * H + j] = + vt::F32ToBF16(table[static_cast((t - m3::kAudioCodeOffset) * H + j)]); + } + } + weights.lm.embed_tokens.bytes = vllm::OwnedBytes(std::move(bytes)); + weights.lm.embed_tokens.dtype = vt::DType::kBF16; + weights.lm.embed_tokens.rank = 2; + weights.lm.embed_tokens.shape[0] = vocab; + weights.lm.embed_tokens.shape[1] = H; + } + + const int32_t semantic_code = 11; + const int64_t frame_index = 2; + const std::vector last_conditional = DeviceArmFill(hidden, &state, 1.0f); + const std::vector last_unconditional = DeviceArmFill(hidden, &state, 1.0f); + + int64_t host_draws = 0; + int64_t device_draws = 0; + const auto make_sampler = [](int64_t* counter) { + return m3::Music3CodeSampler( + [counter](const std::vector& probs, const m3::Music3Draw&) -> int64_t { + ++*counter; + size_t best = 0; + for (size_t i = 1; i < probs.size(); ++i) { + if (probs[i] > probs[best]) best = i; + } + return static_cast(best); + }); + }; + + // The HOST arm, through the same production entry point, with a + // default-constructed arm — which is what every existing caller passes. + std::vector host_codes{semantic_code}; + const std::vector host = m3::Music3DepthStage( + last_conditional, last_unconditional, frame_index, weights, make_sampler(&host_draws), + &host_codes); + + vt::Queue queue = CpuQueue(); + m3::DepthDecoderWeights stage_source = depth; + const m3::Music3DepthDeviceWeights staged = + m3::StageMusic3DepthWeights(queue, config, stage_source, /*release_host=*/false); + m3::Music3DepthDeviceArm arm; + arm.queue = &queue; + arm.depth = &staged; + REQUIRE(arm.engaged()); + REQUIRE_FALSE(arm.half_set()); + + const uint64_t before = m3::Music3DepthDeviceForwardCount(); + std::vector device_codes{semantic_code}; + const std::vector device = + m3::Music3DepthStage(last_conditional, last_unconditional, frame_index, weights, + make_sampler(&device_draws), &device_codes, arm); + const uint64_t after = m3::Music3DepthDeviceForwardCount(); + + // THE ASSERTION #1131 SAYS IS MISSING. `num_codebooks` appends a frame: one + // for the batch-2 prefix at position 0, then one per residual codebook step. + CHECK_MESSAGE(after - before == static_cast(config.num_codebooks), + "the composed stage ran the DEVICE forward " << (after - before) + << " times, expected " + << config.num_codebooks); + REQUIRE(device.size() == host.size()); + double worst_ulp = 0.0; + size_t nonzero = 0; + for (size_t i = 0; i < host.size(); ++i) { + worst_ulp = std::max(worst_ulp, + std::abs(static_cast(device[i]) - host[i]) / Bf16Ulp(host[i])); + if (host[i] != 0.0f) ++nonzero; + } + CHECK_MESSAGE(worst_ulp <= 512.0, "composed device/host worst deviation " + << worst_ulp << " bf16 ULP exceeds 512, which is a " + "STRUCTURAL defect, not rounding"); + // A tolerance CANNOT see a dropped stage — the prefix append, the fed-back + // projection row — because a schedule missing one still produces finite, + // plausible numbers. The drawn codes can. + CHECK_MESSAGE(device_codes == host_codes, "the two arms drew different residual codes"); + CHECK(device_draws == host_draws); + CHECK(device_draws == config.residual_codebooks()); + CHECK(static_cast(device_codes.size()) == config.num_codebooks); + CHECK_MESSAGE(nonzero > 0, "the reference block is all zeros, so nothing is comparable"); + CHECK_MESSAGE(device_draws > 0, "no code was drawn, so the draw order is untested"); + MESSAGE("composed device stage: " << host.size() << " values, worst " << worst_ulp + << " bf16 ULP, " << (after - before) << " device forwards, " + << device_draws << " draws, " << nonzero << " of " + << host.size() << " reference values non-zero"); +} + +TEST_CASE("music3 ar: the device depth arm refuses what it cannot serve, by name") { + const m3::DepthDecoderConfig config = DeviceArmConfig(); + uint32_t state = 0x1234567u; + m3::DepthDecoderWeights depth = DeviceArmWeights(config, &state); + vt::Queue queue = CpuQueue(); + m3::DepthDecoderWeights stage_source = depth; + const m3::Music3DepthDeviceWeights staged = + m3::StageMusic3DepthWeights(queue, config, stage_source, /*release_host=*/false); + const int64_t H = config.hidden_size; + const std::vector row = DeviceArmFill(static_cast(2 * H), &state, 1.0f); + + SUBCASE("a null cache, a non-positive batch and a mis-sized input") { + m3::Music3DepthDeviceCache cache; + CHECK_THROWS_AS(m3::DepthDecoderAppendDevice(queue, config, staged, row, 2, nullptr), + std::runtime_error); + CHECK_THROWS_AS(m3::DepthDecoderAppendDevice(queue, config, staged, row, 0, &cache), + std::runtime_error); + CHECK_THROWS_AS(m3::DepthDecoderAppendDevice(queue, config, staged, row, 3, &cache), + std::runtime_error); + } + SUBCASE("unstaged weights are refused rather than dereferenced") { + m3::Music3DepthDeviceCache cache; + const m3::Music3DepthDeviceWeights empty; + CHECK_FALSE(empty.staged()); + CHECK_THROWS_AS(m3::DepthDecoderAppendDevice(queue, config, empty, row, 2, &cache), + std::runtime_error); + } + SUBCASE("the batch cannot change mid-cache, and the position ceiling binds") { + m3::Music3DepthDeviceCache cache; + m3::DepthDecoderAppendDevice(queue, config, staged, row, 2, &cache); + const std::vector one = DeviceArmFill(static_cast(H), &state, 1.0f); + CHECK_THROWS_AS(m3::DepthDecoderAppendDevice(queue, config, staged, one, 1, &cache), + std::runtime_error); + for (int64_t p = 1; p < config.max_position_embeddings; ++p) { + m3::DepthDecoderAppendDevice(queue, config, staged, row, 2, &cache); + } + CHECK(cache.positions == config.max_position_embeddings); + CHECK_THROWS_AS(m3::DepthDecoderAppendDevice(queue, config, staged, row, 2, &cache), + std::runtime_error); + } + SUBCASE("a mis-sized weight is refused AT STAGE TIME, naming the tensor") { + m3::DepthDecoderWeights broken = depth; + broken.layers[0].gate_proj.pop_back(); + CHECK_THROWS_AS( + m3::StageMusic3DepthWeights(queue, config, broken, /*release_host=*/false), + std::runtime_error); + } + SUBCASE("HALF an arm is refused at the composed stage rather than silently ignored") { + // A caller that set one field thinks it asked for the device and did not. + // Falling back to the host loop would be a 4.4x slowdown wearing a correct + // answer, so it is a refusal. + m3::Music3ArWeights weights; + weights.depth_config = config; + weights.depth = depth; + weights.lm_config.hidden_size = H; + weights.lm_config.vocab_size = m3::kAudioCodeOffset + 8; + m3::Music3DepthDeviceArm queue_only; + queue_only.queue = &queue; + CHECK(queue_only.half_set()); + std::vector codes{1}; + const std::vector last(static_cast(H), 0.5f); + const m3::Music3CodeSampler sampler = + [](const std::vector&, const m3::Music3Draw&) -> int64_t { return 0; }; + CHECK_THROWS_AS(m3::Music3DepthStage(last, last, 0, weights, sampler, &codes, queue_only), + std::runtime_error); + m3::Music3DepthDeviceArm weights_only; + weights_only.depth = &staged; + CHECK(weights_only.half_set()); + std::vector codes2{1}; + CHECK_THROWS_AS(m3::Music3DepthStage(last, last, 0, weights, sampler, &codes2, weights_only), + std::runtime_error); + } +} + TEST_CASE("music3 ar: the audio heads match upstream, one per residual codebook") { const m3::DepthDecoderConfig config = DepthConfig(); const size_t hidden = static_cast(config.hidden_size);