diff --git a/.agents/specs/ltx25-decode-dtype.md b/.agents/specs/ltx25-decode-dtype.md new file mode 100644 index 000000000..de4b29da5 --- /dev/null +++ b/.agents/specs/ltx25-decode-dtype.md @@ -0,0 +1,352 @@ +# LTX25-DECODE-DTYPE — the decode computes in f64, and no reference does + +Row: `LTX25-DECODE-DTYPE`, under the `ROAD-V1-LTX25` campaign +([`roadmap_v1.md`](../roadmap_v1.md), [`ltx-2-5.md`](ltx-2-5.md)). +Issue: [#1008](https://github.com/mudler/vllm.cpp/issues/1008). +Parent: lever 2 of the `LTX25-DECODE-SPEED` investigation +([#1006](https://github.com/mudler/vllm.cpp/issues/1006)), which filed this +issue and lists it under `## Owed`. That spec is +`.agents/specs/ltx25-decode-speed.md` on [PR +#1018](https://github.com/mudler/vllm.cpp/pull/1018) and is **not yet on +`main`**, so it is cited by pull request rather than by relative link — a +relative link would dangle until #1018 lands. + +## Now + +`ACTIVE`. This row takes the *dtype* half of lever 2. It does not take the +memory-format half; §5 states the verdict and what blocks it. + +## 0. Scope, and the one thing this row is not + +**In scope.** Every accumulation and every elementwise arithmetic step on the +LTX-2.5 conv video VAE data path moves from `double` to `float`, which is the +dtype upstream actually computes these ops in. Scalar constants — epsilons, +`1/sqrt(C)`, per-channel shift/scale precomputed from two f32 weights — stay +`double`, because upstream's counterparts are Python floats and they cost +nothing. + +**Not in scope, and deliberately so.** The *storage* dtype stays f32. Upstream +stores bf16 (§1), and moving this file to bf16 storage is the production arm +that `ltx2_video_vae.cpp:63-66` already books as phase L6 debt. That is a +different change with a different risk profile, and this row does not take it. +Narrowing the *arithmetic* is separable from narrowing the *storage*, and only +the first can be done without a second dtype for every buffer. + +**Also not in scope:** threading ([#1009](https://github.com/mudler/vllm.cpp/issues/1009)), +the device arm ([#1007](https://github.com/mudler/vllm.cpp/issues/1007)), +`memory_efficient_decode.py` ([#1011](https://github.com/mudler/vllm.cpp/issues/1011)). + +**No speed number.** `dgx.casa` has been unreachable for the whole of this row's +work, so there is no GPU and no large-render host. This row lands the +correctness-preserving change with the numerics pinned and books the magnitude +as owed and unmeasured (§7). An honest "unmeasured" beats an invented number, +and this campaign's own standing rule is that a number quoted often becomes +treated as measured. + +## 1. What dtype upstream actually accumulates in + +The issue and the parent spec both establish that no reference *stores* f64 on +this path. That is true and it is not the question this row had to answer. The +question is what dtype the *accumulation* happens in, which is not the same as +the tensor dtype, and it was settled by running the reference rather than by +reading it. + +**Read.** Every conv in the decoder is a plain `torch.nn.Conv3d` — `CausalConv3d` +builds it at `packages/ltx-core/src/ltx_core/model/video_vae/convolution.py:292-302` +and the single call site for the whole decoder is `:312`, at +`fd4ded7f2d88d3da713abcdd4ad41ecc4a9314ca`. `PixelNorm.forward` is +`torch.mean(x**2, dim=...)` in the activation dtype +(`model/common/normalization.py:37-40`). `_RMSNorm2D.forward` is +`F.normalize(x, dim=1) * (scale * gamma)`, and `AttnBlock3D.forward` is +`to_qkv` / SDPA / `proj`, all in the activation dtype +(`model/video_vae/attention.py:23, 58-69`). `diffusers` at `3a2f35d4e` agrees: +`PerChannelRMSNorm.forward` computes in the activation dtype +(`src/diffusers/models/autoencoders/autoencoder_kl_ltx2.py:50-59`). + +**Run.** Reading source gives the tensor dtype, not the accumulator width. The +accumulator was measured directly with torch 2.11.0, on a reduction engineered +so the two widths are separable: 27 taps over a uniform-1.0 input with weights +`[+1e8, 0.1 x 25, -1e8]`. Every partial sum `1e8 + j*0.1` for `j <= 25` is below +half an ulp of `1e8` (which is 4.0), so an f32 accumulator holds exactly `1e8` +until the final `-1e8` and lands on exactly zero, in **any** summation order. An +f64 accumulator lands on 2.5. + +| what | result | +|---|---| +| `F.conv3d`, f32 tensors | **0.0** | +| `F.conv3d`, bf16 tensors — upstream's own configuration | **0.0** | +| `F.conv3d`, f64 tensors | 2.500000014901161 | +| naive serial f32, by hand | 0.0 | +| this port today | 2.5 | + +**So upstream's convolution accumulator is f32, and this port's is f64.** The +expected value this row's new gate asserts is therefore torch's own answer, not +a recording of what the patched code happens to emit. + +**One honest caveat, recorded because it bounds what may be gated.** The same +probe run through `torch.sum` / `torch.mean` on f32 returns `2.0999999046325684`, +not `0.0`. Torch's reductions are f32-*wide* but use a cascading/pairwise order, +where this port's are naive serial. Width matches; order does not. A gate that +pinned an adversarial value through `PixelNorm` would therefore be pinning this +port's summation order rather than upstream's dtype, so this row does not build +one. The convolution is the site where torch's answer and a naive serial f32 +answer coincide exactly, and it is also where 42 of the decode's convolutions +and essentially all of its arithmetic live. + +## 2. The sites, and what each one becomes + +The issue counts 8 `double acc` declarations and 29 `static_cast`. The +`acc` count is right and is not the whole set: `PixelNorm`'s `mean_sq`, the +attention block's `sum_sq`, `dot` and `sum`, and the attention block's six +`std::vector` activation scratch buffers are accumulators and data-path +buffers that the `double acc` grep does not reach. + +Anchors are at `ff264cb82`, this row's base. + +| site | what it is | upstream | this row | +|---|---|---|---| +| `:165` | `CausalConv3d` output accumulator — 42 convs, ~all the FLOPs | `nn.Conv3d`, `convolution.py:292-302,312`; measured f32 (§1) | `float` | +| `:201` | `Linear3d`, the 1x1x1 conv used as `conv_shortcut` | `make_linear_nd` dims=3 -> `nn.Conv3d`, `convolution.py:84-85` | `float` | +| `:221` | `PixelNorm` `mean_sq` | `torch.mean(x**2)` in activation dtype, `normalization.py:37-40` | `float` | +| `:303`, `:312` | `TimestepEmbedding`'s two `nn.Linear` accumulators | `nn.Linear` in activation dtype | `float` | +| `:528` | attention `_RMSNorm2D` `sum_sq` | `F.normalize(x, dim=1)`, `attention.py:23` | `float` | +| `:546` | attention `to_qkv` accumulator | 1x1 `nn.Conv2d`, `attention.py:55` | `float` | +| `:557`, `:564` | attention score `dot` and softmax `sum` | SDPA in activation dtype, `attention.py:65` | `float` | +| `:570` | attention value-weighted accumulator | SDPA | `float` | +| `:579` | attention `proj` accumulator | 1x1 `nn.Conv2d`, `attention.py:56` | `float` | +| `:916` | encoder `SpaceToDepthDownsample` group mean | `.mean(...)` in activation dtype | `float` | + +Elementwise data-path steps narrow with them, for the same reason and the same +anchor: `Silu` (`:213`, `F.silu`), the ada-LN apply (`:361-362`), the noise +blend and per-channel denormalize (`:629-631`, `:645-648`), +`FeedSpatialNoise` (`:337-338`), and the encoder's normalize (`:1129-1130`). + +**What stays `double`, each with its reason written beside it in the file.** + +* Stabilizing epsilons (`norm_eps`, `pixel_norm_eps`, `kLtx2RmsNorm2dEps`) — + upstream's are Python floats, they are compared and added once per row, and + narrowing them changes a threshold rather than a data path. +* `norm_scale` = `sqrt(C)` and `attn_scale` = `1/sqrt(C)` (`:514-515`) — + upstream's `channels**0.5` is a Python float, evaluated once per block. +* The `TimestepEmbedding` frequency table (`:284-292`) — a transcendental + constant precompute, not a data-path accumulation, evaluated once per block + over 256 entries. Upstream's `torch.arange`-based table is f32; keeping f64 + here is *closer* to the value that table approximates, and it is not on any + hot path. It is annotated as a deliberate exception rather than left silent. + +## 3. The gate the goldens cannot be + +`ltx2_video_vae.cpp:41-44` already says why, and this row confirmed it rather +than inheriting it: `scripts/gen-ltx2-vae-goldens.py:223` casts every upstream +parameter with `values.astype(np.float32)`, and the input builders at `:759` and +`:836` do the same. **The oracle that produced the goldens ran f32 end to end.** +A dtype comparison against it cannot see an accumulator that is too wide, which +is the defect this row removes. + +That cuts both ways, and the second way is the useful one. Because the golden +generator ran torch in f32, the goldens are the output of an **f32-accumulating** +reference. This port's f64 accumulation has been *wider than the oracle its own +goldens came from* since the file landed. Narrowing to f32 moves this port +toward the goldens' generator, not away from it — which is why §6 expects the +recorded `max|diff|` to hold or improve, and treats a regression as a finding. + +**The new gate.** `test_ltx2_vae.cpp` gains one case that enters through the +production entry point `Ltx2VideoDecodeStreaming` and asserts the decode's +convolution accumulator is f32-wide, using §1's separable reduction. The +decoder is configured so the assertion is analytically derivable end to end +rather than recorded: + +* `decoder_blocks` empty, `timestep_conditioning=false`, `norm_layer=pixel_norm`, + `patch_size=1`, `out_channels=1`, `base_channels=2`, `in_channels=1`, + spatial padding `replicate`, latent all `1.0`, `std-of-means=1`, + `mean-of-means=0`. Every conv tap therefore sees exactly `1.0`, including at + every border, so one reduction is repeated at every output voxel. +* `conv_in` channel 0 carries `[+1e8, 0.1 x 25, -1e8]` with bias 0; channel 1 + carries all-zero weights with bias 1. +* f32 accumulator: ch0 = 0, ch1 = 1 -> `PixelNorm` leaves ch0 at 0 -> `SiLU(0)=0` + -> `conv_out` selects ch0 -> **every output element is exactly 0**. +* f64 accumulator: ch0 = 2.5, ch1 = 1 -> `PixelNorm` inv = `1/sqrt(3.625)` -> + ch0 = 1.3131 -> `SiLU` -> ~1.0348. The two arms are three decimal orders apart. + +The value `0` is upstream's measured answer for this reduction in both f32 and +bf16 (§1), so the case gates a mirrored property and not a local convention. + +## 4. Risks + +* **The narrowing moves a golden past `kLtx2GoldenTol` = 5e-6.** This is the one + risk that can change the design. Naive serial f32 is a worse summation order + than torch's cascading f32, so the error against the goldens can rise even + though the width now matches. Mitigation: measure `max|diff|` per golden arm + before and after, and report both. If an arm exceeds tolerance, the finding is + reported and the site is either kept `double` with its reason written beside it + or given a better summation order — never a widened tolerance. Widening a + tolerance to admit a change is the failure mode `AGENTS.md` forbids. +* **A `float` accumulator silently re-promoted.** `acc += float * double` promotes + the whole expression back to `double`. Every narrowed site must have its scalar + operands narrowed too, or the change is a no-op that reads as done. The §3 gate + catches this for the convolution; the reviewer should mutate the others. +* **`std::exp` overload selection.** `std::exp(-v)` with `float v` selects the + float overload; `std::exp(-static_cast(v))` does not. `Silu` depends on + this. + +## 5. NDHWC — the verdict, and why this row does not build it + +**The engine can express the layout. The blocker is not `Volume`, and naming it +as `Volume` would be wrong.** + +`Volume` is a file-local struct in an anonymous namespace +(`ltx2_video_vae.cpp:85-93`) whose `At()` is a single indexing function with 16 +call sites. That much is cheap. What is not cheap: + +1. **A shared helper hard-codes NCDHW in its signature.** + `MiniMaxH3GroupNorm3d(std::vector& x, int64_t channels, int64_t spatial, ...)` + (`include/vllm/model_executor/models/minimax_h3.h:756`) takes a + channel-major buffer by contract. The LTX-2.5 video VAE calls it at four + sites, and it is **shared**: MiniMax-H3's own VAE CNN calls it at three + (`minimax_h3_vae_cnn.cpp:176,182,337`) and the LTX-2 audio VAE at one + (`ltx2_audio_vae.cpp:215`). NDHWC in the video VAE means either an NDHWC + entry point on that shared helper, or a transpose at every norm boundary + which spends what the layout was meant to save. This is a shared-seam + decision, not a local one. +2. **Ten layout-dependent sites bypass `At()`** with open-coded `c * n + i` + arithmetic, at §2's `ff264cb82` base anchors rather than this section's + final-tree ones — `Linear3d` (`:203,:206`), `ApplyAdaLn` (`:361-362`), the + denormalize (`:645-648`), `SpaceToDepthDownsample` (`:919,:921`), the + encoder normalize (`:1129-1130`) — plus the local scratch buffers `padded`, + `normed`, `q`, `k`, `v` and `attended`, which carry their own layouts and + would each need one. +3. **`Ltx2VideoFrames::data` must not move.** It is the decode's output contract + and the frame writer's input. + +**And the win would not be the layout.** Upstream's `channels_last_3d` +(`memory_efficient_decode.py:617-627`, `:655-656`) exists to select a different +cuDNN 3-D convolution kernel family on a GPU. This port has no device arm at all +([#1007](https://github.com/mudler/vllm.cpp/issues/1007)), so on the host arm +that exists today NDHWC buys nothing by itself: it makes the reduction over +input channels contiguous, which is worth having **only once something +vectorizes over it**. The layout is a precondition for a SIMD or device arm, not +a speedup on a scalar loop nest. + +**Verdict: real, medium-sized, and a separate row.** It is filed rather than +half-built, and it must land after or with the arm that consumes it. The parent +spec's §8 says why bundling it here would be wrong: a dtype change no golden can +see and a memory-format change across a shared seam are two independent reviews. + +## 6. Gates and evidence + +1. **The new width case is RED before the change and GREEN after**, captured + with its exit code, not with a grep of the assertion line. +2. **Reachability.** The case enters through `Ltx2VideoDecodeStreaming`, the + entry `src/vllm/multimodal/ltx2_video.cpp:3258` calls on the render path. + Deleting the `Ltx2ConvVideoDecode` call at + `ltx2_video_vae_tiled.cpp:113` must turn it RED. +3. **`max|diff|` recorded for every video golden arm, before and after.** The + goldens must stay green on their own tolerance, and the two numbers are + reported side by side rather than summarized as "still passes". +4. **Full gate**: configure, build, `ctest`, with `: error:` count, test count, + exit codes, load and free disk. +5. Every mutation reports three facts: `git diff --stat` after applying, whether + it BUILT with the compile-error count, and the exit code. + +## 7. Owed + +| owed | what would settle it | +|---|---| +| **The speed magnitude of this change. UNMEASURED.** No number is claimed. | One `Ltx2ConvVideoDecode` wall at a fixed size on an idle host, same binary, f64 arm against f32 arm. `dgx.casa` was unreachable for this row's whole duration; any host that can run the decode settles it, and #1010's phase timings make it readable from a render. | +| **A width gate for the nine sites that have none** (§8.2). `Linear3d` was widened back to `double` and **40/40 cases passed**. | One separable-reduction case per site, in the shape §3 established for the convolution. | +| **A blocked summation order for the sites that still sum naively** (§8.1) — `Linear3d`, the attention block, `PixelNorm`. `CausalConv3d` has one; the others do not, and their reductions are shorter but not short. | The same treatment, measured the same way. None of them is near tolerance today. | +| NDHWC / `channels_last_3d` (§5) | its own row, after or with a SIMD or device arm | +| bf16 *storage*, the phase L6 production arm (`ltx2_video_vae.cpp:63-66`) | [#1007](https://github.com/mudler/vllm.cpp/issues/1007) and the L6 row | +| The `.agents/issue-index.md` row for [#1008](https://github.com/mudler/vllm.cpp/issues/1008) | It is **deliberately not appended here.** The row exists on PR #1018, which filed the issue and is unmerged. `.gitattributes:7` sets `merge=union` on that file and `scripts/check-agent-record.py:1437-1442` refuses a duplicate issue number with "duplicate is what two branches appending the same issue look like". Appending it here would turn `main` red for every branch the moment #1018 merges — the exact failure a duplicate #995 row caused. The link lives in this spec and in the pull request body; the index link arrives with #1018. | + +## 8. Outcome — what was measured + +### 8.1 The numerics moved, and by how much + +They did not stay put, and §4 named this as the risk that could change the +design. Both arms were run with `kLtx2GoldenTol` temporarily set to `0.0` so +every case reports its `max|diff|` rather than only its verdict. Same binary +recipe, same host, one build each. + +| golden arm | f64 acc (before) | f32 acc, NAIVE serial | f32 acc, BLOCKED (shipped) | tol | +|---|---|---|---|---| +| Conv video decoder | 1.40071e-06 | 4.12762e-06 | **1.72853e-06** | 5e-06 | +| non-causal Conv video decoder | 1.81794e-06 | 3.51667e-06 | 2.08616e-06 | 5e-06 | +| norm_eps-binding video decoder | 9.05246e-07 | 1.16974e-06 | 1.54972e-06 | 5e-06 | +| tiled decode, untiled control A | 2.08616e-06 | **5.00679e-06 FAIL** | 2.74181e-06 | 5e-06 | +| tiled decode, untiled control B | 2.62260e-06 | (same case) | 2.80142e-06 | 5e-06 | +| cropped video encoder | 4.17233e-07 | 8.94070e-07 | 4.76837e-07 | 5e-06 | +| video encoder (`*_res`) | 4.17233e-07 | 8.94070e-07 | 4.76837e-07 | 5e-06 | +| causal-arm video encoder | 2.98023e-07 | 3.83705e-07 | 4.17233e-07 | 5e-06 | +| video encoder (strided convs) | 5.96046e-07 | 5.96046e-07 | 8.34465e-07 | 5e-06 | +| all 13 audio arms | unchanged | unchanged | unchanged | untouched by this row | + +**The risk §4 named as design-changing actually bound, and it changed the +design.** Narrowing the width while keeping the naive serial summation order +pushed `test_ltx2_tiling`'s non-causal untiled control to **5.00679e-06 against a +5e-06 tolerance** — over by 0.14%, a genuine RED in the full gate, not a near +miss. The tolerance was not touched. + +**The fix is the summation ORDER, and it is a closer mirror rather than a looser +one.** `CausalConv3d` now keeps one partial sum per input channel and adds the +partials, so a `ci * kernel^3` reduction accumulates error with `sqrt(kernel^3)` +per block instead of `sqrt(ci * kernel^3)` across the whole length. That is what +torch's f32 convolution does — it is a blocked GEMM, which is exactly why §1's +probe found `torch.sum` returning 2.0999999 where a naive serial f32 sum returns +0.0. The width and the order are two separate mirroring questions and this row +had to answer both. + +**The result is that the numerics essentially did not move.** Against the f64 arm +the shipped blocked-f32 arm is 1.07x to 1.31x on every video arm, where the naive +arm was 1.9x to 2.9x. The worst arm sits at 56% of tolerance — a 1.8x margin, +against 1.9x for the f64 arm it replaces and 1.0x for the naive attempt. No +headroom was meaningfully spent, and no tolerance was widened. + +### 8.2 What is gated, and what is not + +Stated plainly because the answer is uneven and a summary would hide it. Each +narrowed site was widened back to `double` on its own, rebuilt, and rerun. + +| mutation | built | exit | detected by | +|---|---|---|---| +| W1 — `CausalConv3d` accumulator widened | yes, 0 errors | 1 | the new width case (1.03473 vs 7) | +| W2 — `Linear3d` accumulator widened | yes, 0 errors | **0** | **nothing. 40/40 cases pass** | +| W3 — `PixelNorm` `mean_sq` widened | yes, 0 errors | 1 | the Conv video decoder golden, incidentally | +| R — the production call site deleted | yes, 0 errors | 1 | the new width case (7 vs 7-minus-nothing) | + +**W2 is an honest gap and it is owed.** Only the convolution's width is gated on +purpose; `PixelNorm` is caught by accident, because a mixed-width path happens to +diverge from torch further than a uniformly f32 one; and `Linear3d`, the +attention block's four accumulators, the timestep Linears and the encoder's group +mean have no gate on their width at all. They are narrowed on upstream grounding +and on review, not on a test. Closing that needs one separable-reduction case per +site, in the shape §3 established. + +### 8.3 The reachability case that passed while measuring nothing + +Worth recording because it nearly shipped. The first draft of the width case +expected **zero**, which is what an f32 accumulator produces on the engineered +reduction. Mutation R replaced the production `Ltx2ConvVideoDecode` call with a +zero-filled buffer — and the case **passed**, because a decode that never ran +produces zeros too. The case measured nothing and reported success. + +The fix is `conv_out.conv.bias = 7`, which moves the expectation off zero: the +f32 arm must return exactly 7, the f64 arm returns 8.03473, and a stub returns 0. +All three are now distinguishable. A recorded value is not a reached one, and an +expectation that coincides with the zero value of an absent computation is not an +assertion. + +The first attempt at mutation R also **failed to build** (3 `-Werror` unused- +parameter errors) while the stale binary still ran and printed a plausible +verdict. Both facts are why every mutation above reports whether it built. + +## 9. Stop conditions + +* Report `NEEDS_DECISION` rather than widening `kLtx2GoldenTol` if a narrowed + site pushes a golden past it. **This fired** (§8.1). It was resolved by + mirroring upstream's summation order, which this spec had already named as the + remedy, rather than by touching the tolerance or restoring the f64 width — so + it is recorded here as a stop condition that triggered and was answered within + the row's own design, not as one that was waived. +* Do not claim any wall-clock or throughput result. There is no host. +* Do not build the NDHWC change here (§5). diff --git a/docs/FEATURES.md b/docs/FEATURES.md index d5784a9dc..1163a0764 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -168,6 +168,7 @@ in `ltx2_text_encoder.cpp` is the call that would have to change. | MiniMax-Music3 (`MiniMaxMusic3ForConditionalGeneration`, diffusers lane) | MiniMax-Music3 (8.6B Qwen3 LLM + 0.646B RVQ decoder + 2.4B fp32 DiT + DAC Flow-VAE); diffusers arm, ~28.5 GB | `ACTIVE`. Loader 1413/1413; AR, acoustic and the 8.6B LM forward all gated vs real weights; `SpeechRegistry` + `vllm_speech_*` v20 + `/v1/audio/speech`; GGUF Q4_K depth decoder value-gated. HTTP request OBSERVED (#852) | Not measured. The denominator will be SGLang-Omni in its production configuration (both CUDA graphs, compiled DIT and DAV, batched seeded sampling) | | LTX-2.5 DFR base + generated keyframe slots | LTX-2.5 (21.00B video+audio) | gated vs EXECUTED upstream `dfr_layout` + 3 `dfr_pipeline` helpers @ `fd4ded7f` (`test_ltx2_dfr` 11/11, 652 assertions); canvas, tiles, stitch, carry-forward as EXACT index vectors, since each defect is plausible| `--pipeline-kind dfr`. Canvas PADS 9 to 25 then trims back; slots on the x8 grid, MARKED, read back BEFORE the trim. `num_generated_keyframes` SERVED elsewhere. Temporal ROUNDS refused (#986); detail LoRA refused (#975)| | LTX-2.5 tiled + streaming Conv VAE decode | LTX-2.5 video VAE | gated vs executed upstream `ltx_core` @ `fd4ded7f` (`test_ltx2_tiling` 10/10, 915 assertions); one-tile and untiled-spatial controls BIT-EXACT vs untiled on both causality arms; an untiled frames axis is REFUSED | Streams temporal chunks through upstream's AUTO layout (768/64 px, 80/24 frames); above one tile the pixel volume is never materialized. NO-OP below 768px and 81 frames; 81-120 IS tiled, differing 6.70% of range | +| LTX-2.5 Conv VAE decode arithmetic width | LTX-2.5 video VAE | `test_ltx2_vae` "the decode's convolution accumulates in f32", entering through `Ltx2VideoDecodeStreaming`; widening the accumulator to `double`, or deleting the production call site, each turns it RED | **f32**, the width `F.conv3d` uses at f32 AND bf16 (MEASURED). Was f64 at 8 sites ([#1008](https://github.com/mudler/vllm.cpp/issues/1008)). Conv sums BLOCKED per input channel, as torch's. STORAGE stays f32; bf16 owed | | LTX-2.5 retake (`RetakePipeline`, regenerate a time window) | LTX-2.5 DiT + video VAE encoder | `test_ltx2_retake` 4/4 (69 assertions) and 4 `test_ltx2_video` cases entering through `Generate`; mask, conform and the four-way plan pinned to upstream `fd4ded7f` | `--pipeline-kind retake` on `ltx2-gen`. Source is a `frame_%06d.ppm` DIRECTORY; a container is REFUSED (no demuxer). Geometry comes from the clip. A folder has no audio, so the soundtrack is generated | | MTP speculator | Qwen3.6-27B, Qwen3.6-35B-A3B | token-identical to vLLM `mtp` at c1 | ~4% faster c1; +16% output tput (MoE) | | DFlash block-diffusion | Qwen3 (DFlash draft) | near-tie e2e 27/27 vs vLLM | 2.9x over spec-off, 1.003x vs vLLM DFlash-on | diff --git a/src/vllm/model_executor/models/ltx2_video_vae.cpp b/src/vllm/model_executor/models/ltx2_video_vae.cpp index ad7880b72..dafc39b9c 100644 --- a/src/vllm/model_executor/models/ltx2_video_vae.cpp +++ b/src/vllm/model_executor/models/ltx2_video_vae.cpp @@ -43,6 +43,23 @@ // every upstream parameter to f32, so the oracle itself runs f32 and a dtype // comparison against it is vacuous by construction. // +// ─── THE ARITHMETIC IS f32 TOO, AND USED NOT TO BE (#1008) ─────────────────── +// Storage being f32 says nothing about the width the arithmetic runs at, and +// until #1008 this file accumulated every convolution, GEMM, norm and softmax in +// `double` — a width no reference uses anywhere on this path. Upstream's ops are +// plain `nn.Conv3d` / `nn.Conv2d` / `F.normalize` / SDPA, which accumulate in the +// tensor dtype. That was MEASURED rather than assumed: on a reduction engineered +// so the widths separate, `F.conv3d` returns 0.0 for f32 AND for bf16 tensors +// while an f64 accumulator returns 2.5. The case +// "the decode's convolution accumulates in f32" in tests/vllm/models/ +// test_ltx2_vae.cpp is that instrument, and it is the only gate here that can +// see the width — for the reason the paragraph above gives. +// +// What deliberately stays f64, each annotated at its site: the pinned config +// epsilons, the once-per-block scalars `sqrt(C)` and `1/sqrt(C)`, and the +// TimestepEmbedding frequency table, which is a constant precompute rather than +// a data path. +// // PHASE L6 OWES THE PRODUCTION ARM — the bf16/NVFP4 decode that inherits the // checkpoint dtype the way upstream does. Until it lands, this file is a // correctness reference, not the shipping path, and no memory or throughput @@ -162,23 +179,38 @@ Volume CausalConv3d(const Volume& in, int64_t out_channels, int64_t kernel, bool for (int64_t ti = 0; ti < out.t; ++ti) { for (int64_t hi = 0; hi < out.h; ++hi) { for (int64_t wi = 0; wi < out.w; ++wi) { - double acc = bias != nullptr ? (*bias)[static_cast(oc)] : 0.0; + // f32, because that is the width `nn.Conv3d` accumulates in — MEASURED, + // not assumed: F.conv3d returns 0.0 on the separable reduction in + // tests/vllm/models/test_ltx2_vae.cpp for f32 AND for bf16 tensors, + // where an f64 accumulator returns 2.5 (#1008). + float acc = bias != nullptr ? (*bias)[static_cast(oc)] : 0.0f; for (int64_t ic = 0; ic < ci; ++ic) { + // BLOCKED, one partial sum per input channel, and this is the ORDER + // as well as the width. A single naive serial f32 sum over all + // `ci * kernel^3` taps accumulates error with sqrt of the whole + // length; splitting it into `ci` blocks of `kernel^3` accumulates + // with sqrt of each. That is not a local optimisation — torch's f32 + // convolution is a blocked GEMM and sums exactly this way, which is + // why `torch.sum` on the separable reduction returns 2.0999999 where + // a naive serial f32 sum returns 0.0. Narrowing the width alone, + // with the naive order kept, pushed the non-causal tiled golden to + // 5.00679e-06 against a 5e-06 tolerance — MEASURED, and the reason + // this loop is shaped this way. + float tap = 0.0f; for (int64_t a = 0; a < kernel; ++a) { for (int64_t b = 0; b < kernel; ++b) { for (int64_t d = 0; d < kernel; ++d) { - acc += static_cast( - padded[static_cast(((ic * pt + ti * stride_t + a) * ph + - hi * stride_h + b) * - pw + - wi * stride_w + d)]) * - static_cast(weight[static_cast( - (((oc * ci + ic) * kernel + a) * kernel + b) * kernel + d)]); + tap += padded[static_cast( + ((ic * pt + ti * stride_t + a) * ph + hi * stride_h + b) * pw + + wi * stride_w + d)] * + weight[static_cast( + (((oc * ci + ic) * kernel + a) * kernel + b) * kernel + d)]; } } } + acc += tap; } - out.data[out.At(oc, ti, hi, wi)] = static_cast(acc); + out.data[out.At(oc, ti, hi, wi)] = acc; } } } @@ -198,19 +230,25 @@ Volume Linear3d(const Volume& in, int64_t out_channels, const std::vector const int64_t n = in.spatial(); for (int64_t oc = 0; oc < out_channels; ++oc) { for (int64_t i = 0; i < n; ++i) { - double acc = bias[static_cast(oc)]; + // f32: this is an `nn.Conv3d` upstream too (make_linear_nd's dims==3 + // branch, convolution.py:84-85), so it accumulates at the same width as + // every other conv on the path. + float acc = bias[static_cast(oc)]; for (int64_t ic = 0; ic < in.channels; ++ic) { - acc += static_cast(in.data[static_cast(ic * n + i)]) * - static_cast(weight[static_cast(oc * in.channels + ic)]); + acc += in.data[static_cast(ic * n + i)] * + weight[static_cast(oc * in.channels + ic)]; } - out.data[static_cast(oc * n + i)] = static_cast(acc); + out.data[static_cast(oc * n + i)] = acc; } } return out; } void Silu(std::vector& x) { - for (float& v : x) v = static_cast(v / (1.0 + std::exp(-static_cast(v)))); + // f32: `F.silu` computes in the activation dtype. `std::exp(-v)` on a float + // selects the float overload — spelling it `-static_cast(v)` would + // quietly restore the f64 path this deliberately leaves. + for (float& v : x) v = v / (1.0f + std::exp(-v)); } // PixelNorm() with its DEFAULT eps of 1e-8 (normalization.py:22, reached bare @@ -218,16 +256,22 @@ void Silu(std::vector& x) { // audio VAE gets through build_normalization_layer. void PixelNorm(std::vector& x, int64_t channels, int64_t spatial, double eps) { for (int64_t i = 0; i < spatial; ++i) { - double mean_sq = 0.0; + // f32: `torch.mean(x**2, dim=...)` runs in the activation dtype + // (normalization.py:37-40). + float mean_sq = 0.0f; for (int64_t c = 0; c < channels; ++c) { - const double v = x[static_cast(c * spatial + i)]; + const float v = x[static_cast(c * spatial + i)]; mean_sq += v * v; } - mean_sq /= static_cast(channels); - const double inv = 1.0 / std::sqrt(mean_sq + eps); + mean_sq /= static_cast(channels); + // The reciprocal is f32 too: upstream is `x / torch.sqrt(mean_sq + eps)` + // with every term in the tensor dtype (normalization.py:37-40). `eps` + // remains an f64 PARAMETER because it is a pinned config threshold + // (Ltx2ConvVideoDecoderConfig::pixel_norm_eps); it is narrowed here, at the + // one point it enters the arithmetic. + const float inv = 1.0f / std::sqrt(mean_sq + static_cast(eps)); for (int64_t c = 0; c < channels; ++c) { - x[static_cast(c * spatial + i)] = - static_cast(x[static_cast(c * spatial + i)] * inv); + x[static_cast(c * spatial + i)] = x[static_cast(c * spatial + i)] * inv; } } } @@ -281,6 +325,12 @@ std::vector TimestepEmbedding(double timestep, int64_t embedding_dim, constexpr int64_t kProjChannels = 256; constexpr double kMaxPeriod = 10000.0; const int64_t half = kProjChannels / 2; + // DELIBERATE f64 EXCEPTION, and the only one on this path. `proj` is a + // transcendental CONSTANT table — 256 cos/sin values built once per block from + // the timestep, never a per-element data-path accumulation — so it is off + // every hot path, and evaluating it in f64 sits closer to the exact value that + // upstream's f32 `torch.arange` table approximates. Everything downstream of + // it is f32 (#1008). std::vector proj(static_cast(kProjChannels)); for (int64_t i = 0; i < half; ++i) { // downscale_freq_shift = 0, so the divisor is exactly half_dim. @@ -298,23 +348,25 @@ std::vector TimestepEmbedding(double timestep, int64_t embedding_dim, VT_CHECK(static_cast(w1.size()) == embedding_dim * kProjChannels, "ltx2 timestep embedding: linear_1 shape does not match the embedding dim"); - std::vector hidden(static_cast(embedding_dim)); + // f32 for both `nn.Linear` accumulators and for the hidden activation between + // them: upstream's TimestepEmbedder is two plain Linears with a SiLU, all in + // the activation dtype. The frequency table above stays f64 — see its note. + std::vector hidden(static_cast(embedding_dim)); for (int64_t o = 0; o < embedding_dim; ++o) { - double acc = b1[static_cast(o)]; + float acc = b1[static_cast(o)]; for (int64_t i = 0; i < kProjChannels; ++i) { - acc += proj[static_cast(i)] * - static_cast(w1[static_cast(o * kProjChannels + i)]); + acc += static_cast(proj[static_cast(i)]) * + w1[static_cast(o * kProjChannels + i)]; } - hidden[static_cast(o)] = acc / (1.0 + std::exp(-acc)); // SiLU + hidden[static_cast(o)] = acc / (1.0f + std::exp(-acc)); // SiLU } std::vector out(static_cast(embedding_dim)); for (int64_t o = 0; o < embedding_dim; ++o) { - double acc = b2[static_cast(o)]; + float acc = b2[static_cast(o)]; for (int64_t i = 0; i < embedding_dim; ++i) { - acc += hidden[static_cast(i)] * - static_cast(w2[static_cast(o * embedding_dim + i)]); + acc += hidden[static_cast(i)] * w2[static_cast(o * embedding_dim + i)]; } - out[static_cast(o)] = static_cast(acc); + out[static_cast(o)] = acc; } return out; } @@ -330,12 +382,12 @@ void FeedSpatialNoise(Volume& x, const std::vector& per_channel_scale, VT_CHECK(static_cast(plane.size()) == x.h * x.w, "ltx2 video vae: the noise stream returned the wrong element count"); for (int64_t c = 0; c < x.channels; ++c) { - const double scale = per_channel_scale[static_cast(c)]; + // f32: upstream scales and adds the noise plane in the activation dtype. + const float scale = per_channel_scale[static_cast(c)]; for (int64_t ti = 0; ti < x.t; ++ti) { for (int64_t hi = 0; hi < x.h; ++hi) { for (int64_t wi = 0; wi < x.w; ++wi) { - x.data[x.At(c, ti, hi, wi)] += static_cast( - static_cast(plane[static_cast(hi * x.w + wi)]) * scale); + x.data[x.At(c, ti, hi, wi)] += plane[static_cast(hi * x.w + wi)] * scale; } } } @@ -353,13 +405,15 @@ void ApplyAdaLn(Volume& x, const std::vector& table, const std::vector(table[static_cast(shift_row * c + ch)]) + - static_cast(embed[static_cast(shift_row * c + ch)]); - const double scale = static_cast(table[static_cast(scale_row * c + ch)]) + - static_cast(embed[static_cast(scale_row * c + ch)]); + // f32: upstream adds two f32 tensors and applies `x * (1 + scale) + shift` + // in the activation dtype (resnet.py:135-147). + const float shift = table[static_cast(shift_row * c + ch)] + + embed[static_cast(shift_row * c + ch)]; + const float scale = table[static_cast(scale_row * c + ch)] + + embed[static_cast(scale_row * c + ch)]; for (int64_t i = 0; i < n; ++i) { - x.data[static_cast(ch * n + i)] = static_cast( - static_cast(x.data[static_cast(ch * n + i)]) * (1.0 + scale) + shift); + x.data[static_cast(ch * n + i)] = + x.data[static_cast(ch * n + i)] * (1.0f + scale) + shift; } } } @@ -515,73 +569,87 @@ Volume AttnBlock3d(const Ltx2VaeWeights& weights, const std::string& prefix, con const double attn_scale = 1.0 / std::sqrt(static_cast(c)); Volume out = x; - std::vector normed(static_cast(c * n)); - std::vector q(static_cast(c * n)), k(static_cast(c * n)), + // f32 activations, not f64. Upstream holds q/k/v and the attention output in + // the tensor dtype (attention.py:63-67) and never promotes; these six buffers + // are the block's whole scratch footprint, so the width is bytes as well as + // arithmetic. `norm_scale` and `attn_scale` above stay f64 — upstream's + // `channels**0.5` is a Python float evaluated once per block. + std::vector normed(static_cast(c * n)); + std::vector q(static_cast(c * n)), k(static_cast(c * n)), v(static_cast(c * n)); - std::vector scores(static_cast(n)); - std::vector attended(static_cast(c * n)); + std::vector scores(static_cast(n)); + std::vector attended(static_cast(c * n)); for (int64_t frame = 0; frame < x.t; ++frame) { // _RMSNorm2D: F.normalize(x, dim=1) * (sqrt(C) * gamma) — an L2 normalize with // torch's 1e-12 floor, not a mean-square RMS. for (int64_t i = 0; i < n; ++i) { - double sum_sq = 0.0; + // f32: `F.normalize(x, dim=1)` computes its norm in the input dtype + // (attention.py:23). torch's 1e-12 floor stays f64 — it is a threshold. + float sum_sq = 0.0f; for (int64_t ch = 0; ch < c; ++ch) { - const double value = x.data[x.At(ch, frame, i / x.w, i % x.w)]; + const float value = x.data[x.At(ch, frame, i / x.w, i % x.w)]; sum_sq += value * value; } - const double inv = 1.0 / std::max(std::sqrt(sum_sq), kLtx2RmsNorm2dEps); + const float inv = static_cast( + 1.0 / std::max(std::sqrt(static_cast(sum_sq)), kLtx2RmsNorm2dEps)); + // Same left-to-right association the f64 arm used; only the width changes. + const float norm_scale_f = static_cast(norm_scale); for (int64_t ch = 0; ch < c; ++ch) { - normed[static_cast(ch * n + i)] = - x.data[x.At(ch, frame, i / x.w, i % x.w)] * inv * norm_scale * - static_cast(gamma[static_cast(ch)]); + normed[static_cast(ch * n + i)] = x.data[x.At(ch, frame, i / x.w, i % x.w)] * inv * + norm_scale_f * gamma[static_cast(ch)]; } } // to_qkv is a 1x1 Conv2d emitting [q | k | v] along the channel axis, and the // rearrange to tokens keeps that split on the LAST axis (attention.py:63-64). + // f32: `to_qkv` is a 1x1 nn.Conv2d (attention.py:55), the same accumulator + // width as every other conv here. for (int64_t oc = 0; oc < 3 * c; ++oc) { - std::vector& dst = oc < c ? q : (oc < 2 * c ? k : v); + std::vector& dst = oc < c ? q : (oc < 2 * c ? k : v); const int64_t row = oc % c; for (int64_t i = 0; i < n; ++i) { - double acc = qkv_b[static_cast(oc)]; + float acc = qkv_b[static_cast(oc)]; for (int64_t ic = 0; ic < c; ++ic) { - acc += normed[static_cast(ic * n + i)] * - static_cast(qkv_w[static_cast(oc * c + ic)]); + acc += normed[static_cast(ic * n + i)] * qkv_w[static_cast(oc * c + ic)]; } dst[static_cast(row * n + i)] = acc; } } + // f32: SDPA computes scores, softmax and the value-weighted sum in the + // tensor dtype (attention.py:65). `attn_scale` stays f64 for the same reason + // `norm_scale` does. + const float attn_scale_f = static_cast(attn_scale); for (int64_t i = 0; i < n; ++i) { - double max_score = -std::numeric_limits::infinity(); + float max_score = -std::numeric_limits::infinity(); for (int64_t j = 0; j < n; ++j) { - double dot = 0.0; + float dot = 0.0f; for (int64_t ch = 0; ch < c; ++ch) { dot += q[static_cast(ch * n + i)] * k[static_cast(ch * n + j)]; } - scores[static_cast(j)] = dot * attn_scale; + scores[static_cast(j)] = dot * attn_scale_f; max_score = std::max(max_score, scores[static_cast(j)]); } - double sum = 0.0; + float sum = 0.0f; for (int64_t j = 0; j < n; ++j) { scores[static_cast(j)] = std::exp(scores[static_cast(j)] - max_score); sum += scores[static_cast(j)]; } for (int64_t ch = 0; ch < c; ++ch) { - double acc = 0.0; + float acc = 0.0f; for (int64_t j = 0; j < n; ++j) { acc += scores[static_cast(j)] * v[static_cast(ch * n + j)]; } attended[static_cast(ch * n + i)] = acc / sum; } } + // f32: `proj` is a 1x1 nn.Conv2d (attention.py:56). for (int64_t oc = 0; oc < c; ++oc) { for (int64_t i = 0; i < n; ++i) { - double acc = proj_b[static_cast(oc)]; + float acc = proj_b[static_cast(oc)]; for (int64_t ic = 0; ic < c; ++ic) { - acc += attended[static_cast(ic * n + i)] * - static_cast(proj_w[static_cast(oc * c + ic)]); + acc += attended[static_cast(ic * n + i)] * proj_w[static_cast(oc * c + ic)]; } - out.data[out.At(oc, frame, i / x.w, i % x.w)] += static_cast(acc); + out.data[out.At(oc, frame, i / x.w, i % x.w)] += acc; } } } @@ -625,10 +693,12 @@ Ltx2VideoFrames Ltx2ConvVideoDecode(const Ltx2ConvVideoDecoderConfig& config, const std::vector drawn = noise->Draw(static_cast(x.data.size())); VT_CHECK(drawn.size() == x.data.size(), "ltx2 video vae: the noise stream returned the wrong element count"); + // f32: the blend runs in the activation dtype upstream. The two scalars are + // config values, so they are narrowed once rather than per element. + const float noise_scale = static_cast(config.decode_noise_scale); + const float keep_scale = static_cast(1.0 - config.decode_noise_scale); for (size_t i = 0; i < x.data.size(); ++i) { - x.data[i] = static_cast( - static_cast(drawn[i]) * config.decode_noise_scale + - (1.0 - config.decode_noise_scale) * static_cast(x.data[i])); + x.data[i] = drawn[i] * noise_scale + keep_scale * x.data[i]; } } { @@ -640,12 +710,13 @@ Ltx2VideoFrames Ltx2ConvVideoDecode(const Ltx2ConvVideoDecoderConfig& config, static_cast(mean_of_means.size()) == latent_channels, "ltx2 video vae: per-channel statistics must have one value per latent channel"); const int64_t n = x.spatial(); + // f32: upstream's de-normalize is `latent * std + mean` on f32/bf16 tensors. for (int64_t c = 0; c < latent_channels; ++c) { + const float std_c = std_of_means[static_cast(c)]; + const float mean_c = mean_of_means[static_cast(c)]; for (int64_t i = 0; i < n; ++i) { - x.data[static_cast(c * n + i)] = static_cast( - static_cast(x.data[static_cast(c * n + i)]) * - static_cast(std_of_means[static_cast(c)]) + - static_cast(mean_of_means[static_cast(c)])); + x.data[static_cast(c * n + i)] = + x.data[static_cast(c * n + i)] * std_c + mean_c; } } } @@ -913,13 +984,12 @@ Volume SpaceToDepthDownsample(const VideoConvSpec& spec, const Ltx2VaeWeights& w const int64_t n = skip.spatial(); for (int64_t c = 0; c < out_channels; ++c) { for (int64_t i = 0; i < n; ++i) { - double acc = 0.0; + // f32: upstream's group mean runs in the activation dtype. + float acc = 0.0f; for (int64_t g = 0; g < group_size; ++g) { - acc += static_cast( - folded_in.data[static_cast((c * group_size + g) * n + i)]); + acc += folded_in.data[static_cast((c * group_size + g) * n + i)]; } - skip.data[static_cast(c * n + i)] = - static_cast(acc / static_cast(group_size)); + skip.data[static_cast(c * n + i)] = acc / static_cast(group_size); } } @@ -1122,12 +1192,14 @@ Ltx2LatentVolume Ltx2ConvVideoEncode(const Ltx2ConvVideoEncoderConfig& config, out.width = x.w; out.data.resize(static_cast(out.elems())); const int64_t elems = x.spatial(); + // f32: the encoder's normalize is the decoder de-normalize run backwards, and + // upstream computes it in the activation dtype on both sides. for (int64_t c = 0; c < latent_channels; ++c) { - const double mean = mean_of_means[static_cast(c)]; - const double denom = std_of_means[static_cast(c)]; + const float mean = mean_of_means[static_cast(c)]; + const float denom = std_of_means[static_cast(c)]; for (int64_t i = 0; i < elems; ++i) { - out.data[static_cast(c * elems + i)] = static_cast( - (static_cast(x.data[static_cast(c * elems + i)]) - mean) / denom); + out.data[static_cast(c * elems + i)] = + (x.data[static_cast(c * elems + i)] - mean) / denom; } } return out; diff --git a/tests/vllm/models/test_ltx2_vae.cpp b/tests/vllm/models/test_ltx2_vae.cpp index 12c8c2e0a..b3f688d41 100644 --- a/tests/vllm/models/test_ltx2_vae.cpp +++ b/tests/vllm/models/test_ltx2_vae.cpp @@ -26,6 +26,9 @@ #include "vllm/model_executor/models/ltx2_audio_vae.h" #include "vllm/model_executor/models/ltx2_audio_vae_encoder.h" #include "vllm/model_executor/models/ltx2_conditioning.h" +// The accumulator-width case enters the decode through the PRODUCTION streaming +// entry point rather than through Ltx2ConvVideoDecode (issue #1008). +#include "vllm/model_executor/models/ltx2_tiling.h" #include "vllm/model_executor/models/ltx2_video_vae.h" #include "vllm/model_executor/models/ltx2_video_vae_encoder.h" // vocoder1d::kSnakeEps: the Snake/SnakeBeta stabilizer is SHARED with MiniMax-H3's @@ -1059,6 +1062,128 @@ TEST_CASE("ltx2 vae: the NON-causal Conv video decoder matches upstream ltx_core CHECK(causal_frames.data != frames.data); } +TEST_CASE("ltx2 vae: the decode's convolution accumulates in f32, the width torch uses") { + // THE ONE DEFECT THE GOLDENS STRUCTURALLY CANNOT REPORT, so it gets its own + // instrument (issue #1008, .agents/specs/ltx25-decode-dtype.md). + // + // Every golden in this file is vacuous on accumulator WIDTH by construction: + // scripts/gen-ltx2-vae-goldens.py:223 casts every upstream parameter with + // `values.astype(np.float32)`, so the oracle ran f32 end to end and a dtype + // comparison against it compares nothing. A decode that accumulated in f64 — + // as this one did until #1008 — stays numerically plausible, keeps every + // golden green, and moves twice the bytes. AGENTS.md says a token gate cannot + // detect a dtype that is too wide; this case is what detects it here. + // + // THE REDUCTION IS ENGINEERED SO THE TWO WIDTHS ARE SEPARABLE, and the + // expected value is UPSTREAM'S, not a recording of what this port emits. + // MEASURED with torch 2.11.0 on exactly the taps below: + // F.conv3d, f32 tensors -> 0.0 + // F.conv3d, bf16 tensors -> 0.0 (upstream's own dtype, distilled.py:109) + // F.conv3d, f64 tensors -> 2.500000014901161 + // Why 0.0 holds for ANY f32 summation order: half an ulp of 1e8 is 4.0, and + // the 25 small taps sum to 2.5, so every partial sum rounds back to exactly + // 1e8 and the closing -1e8 cancels it exactly. An f64 accumulator keeps the + // 2.5. There is no order in which an f32 accumulator does. + constexpr float kBig = 1e8f; + constexpr float kSmall = 0.1f; + + vllm::Ltx2ConvVideoDecoderConfig cfg; + cfg.prefix = "ltx2.videodec.accwidth."; + cfg.in_channels = 1; + cfg.out_channels = 1; + cfg.patch_size = 1; + cfg.base_channels = 2; + cfg.causal = false; + cfg.timestep_conditioning = false; + cfg.norm_layer = vllm::Ltx2NormLayer::kPixelNorm; + // REPLICATE, not the fixtures' reflect: with a uniform input every conv tap + // then reads exactly 1.0 at every output voxel INCLUDING the borders, so one + // reduction is repeated everywhere and no voxel is a special case. `kZeros` + // would zero the border taps and break the derivation below. + cfg.spatial_padding_mode = vllm::Ltx2PaddingMode::kReplicate; + // No blocks at all. The decode is then conv_in -> PixelNorm -> SiLU -> + // conv_out, which is short enough to carry an ANALYTIC expectation end to end + // rather than a checked-in vector that only records today's behaviour. + cfg.decoder_blocks = {}; + + const std::string p = cfg.prefix; + vllm::Ltx2VaeWeights weights; + weights.tensors[p + "per_channel_statistics.std-of-means"] = {1.0f}; + weights.tensors[p + "per_channel_statistics.mean-of-means"] = {0.0f}; + + // conv_in is [out=2, in=1, 3, 3, 3]. Output channel 0 carries the separable + // reduction in the exact order CausalConv3d walks it (ic, then a, b, d, so + // flat 0..26); output channel 1 is all-zero with a bias of 1, purely to give + // PixelNorm a second channel with a known norm. + std::vector conv_in(2 * 1 * 27, 0.0f); + conv_in[0] = kBig; + for (size_t i = 1; i < 26; ++i) conv_in[i] = kSmall; + conv_in[26] = -kBig; + weights.tensors[p + "conv_in.conv.weight"] = conv_in; + weights.tensors[p + "conv_in.conv.bias"] = {0.0f, 1.0f}; + + // conv_out is [out=1, in=2, 3, 3, 3]; it selects channel 0's centre tap and + // nothing else, so it forwards the value under test without adding a second + // reduction that could mask it. + // + // THE BIAS IS 7, AND THAT IS NOT COSMETIC. With a bias of 0 the f32 answer is + // zero, and a decode that never ran — a deleted call site handing back a + // zero-filled buffer — produces zero as well, so the case would pass while + // measuring nothing. This was not hypothetical: the reachability mutation was + // run, the stub returned zeros, and an earlier draft of this case PASSED. + // Offsetting the expectation off zero is what separates "accumulated in f32" + // from "nothing reached this at all". + constexpr float kOutBias = 7.0f; + std::vector conv_out(1 * 2 * 27, 0.0f); + conv_out[13] = 1.0f; // ic=0, a=b=d=1 + weights.tensors[p + "conv_out.conv.weight"] = conv_out; + weights.tensors[p + "conv_out.conv.bias"] = {kOutBias}; + + const int64_t lt = 1, lh = 2, lw = 2; + const std::vector latent(static_cast(lt * lh * lw), 1.0f); + + vllm::Ltx2TileSizeConfig tiling; + tiling.frames = vllm::Ltx2DimensionSizeConfig{10000, 0}; + tiling.height = vllm::Ltx2DimensionSizeConfig{10000, 0}; + tiling.width = vllm::Ltx2DimensionSizeConfig{10000, 0}; + + // ENTERS THROUGH THE PRODUCTION ENTRY POINT. `Ltx2VideoDecodeStreaming` is + // what the render path calls (src/vllm/multimodal/ltx2_video.cpp:3258), and it + // reaches Ltx2ConvVideoDecode through ltx2_video_vae_tiled.cpp:113. Deleting + // that call site turns this case RED, which is the reachability proof; a case + // that called Ltx2ConvVideoDecode directly would prove only that the function + // works. A null noise stream is deliberate: this configuration must not draw, + // and the decode's own VT_CHECK fails loudly if that ever changes. + int64_t chunks = 0; + vllm::Ltx2VideoFrames got; + vllm::Ltx2VideoDecodeStreaming( + vllm::Ltx2VideoDecoderKind::kConv, cfg, weights, latent, cfg.in_channels, lt, lh, lw, + /*noise=*/nullptr, tiling, [&](const vllm::Ltx2VideoChunk& chunk) { + ++chunks; + got = chunk.frames; + }); + REQUIRE(chunks == 1); + REQUIRE(got.data.size() == 4u); + + // THE DERIVATION, so a reader can check the number rather than trust it. + // conv_in ch0 = 0 (f32) or 2.5 (f64); conv_in ch1 = bias = 1 + // PixelNorm over 2 channels: f32 mean_sq = 0.5, and 0 * anything = 0 + // SiLU(0) = 0; conv_out forwards ch0 and adds its bias + // so an f32 accumulator makes EVERY output element exactly the bias. An f64 + // one gives 2.5 -> 2.5/sqrt(3.625) = 1.31306 -> SiLU -> ~1.03473 on top of it, + // a gap of 1.03, which is 200000x the golden tolerance. Nothing between the + // two arms is a matter of tolerance. + const std::vector want(4, kOutBias); + const double err = MaxAbsDiff(got.data, want.data(), got.data.size()); + INFO("conv accumulator width probe max|out - bias| = " << err); + CHECK(err <= kLtx2GoldenTol); + + // The f64 answer stated as a value, so this case fails LOUDLY rather than + // drifting if the accumulator is ever widened again. 1.03473 is what this + // decode returned before #1008 narrowed it. + CHECK(err < 1.0); +} + TEST_CASE("ltx2 vae: the video decoder's norm_eps is gated where it BINDS") { // THE ARM THAT MAKES `Ltx2ConvVideoDecoderConfig::norm_eps` NUMERICALLY // REACHABLE, and the correction of a record that said it was not.