diff --git a/.agents/issue-index.md b/.agents/issue-index.md index 9054038b6..9512f191a 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -326,3 +326,5 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#1092](https://github.com/mudler/vllm.cpp/issues/1092) | `LTX25-GUIDED-VIDEO` | The LTX-2.5 **video** denoise loop runs one UNGUIDED forward per step: `Ltx2PhaseRecipe::video_guidance` is set by every recipe (`src/vllm/model_executor/models/ltx2_pipeline.cpp:1069 @ b5756ea8c`) and read by nothing, so a `pipeline_kind = one_stage` render ignores `cfg_scale = 3.0`, `stg_scale = 1.0`, `rescale_scale = 0.7` and `modality_scale = 3.0` and denoises along a different trajectory than `ti2vid_one_stage.py:221-226 @ fd4ded7f`, which builds a `FactoryGuidedDenoiser` from exactly those. `allow_guidance_override` (`ltx2_pipeline.h:534`) is dead the same way. Positive control for the grep: the same command for `audio_guidance` returns the T2A consumer at `ltx2_video.cpp:3527`. Blocks four more pipelines on one missing seam (`a2vid_two_stage.py:230`, `ti2vid_two_stages.py:248`, `ti2vid_two_stages_hq.py:271`, `keyframe_interpolation.py:232`). Spec [`ltx25-guided-video.md`](specs/ltx25-guided-video.md) | bug | | [#1111](https://github.com/mudler/vllm.cpp/issues/1111) | `LTX25-GUIDED-VIDEO` | `Ltx2T2aGuidedStep` (`src/vllm/model_executor/models/ltx2_t2a.cpp:203-214` @ `420f6b474`) builds the STG block mask and refuses when NO bit is set, and that one predicate catches two different things. One is a request that disagrees with the CHECKPOINT - `[28]` on a two-block DiT - which is a LOCAL condition, because upstream only ever runs 48-block checkpoints and this port runs reduced ones. The other is an EMPTY list, which is upstream's OWN documented spelling for "STG off" and is refused here. Measured at Lightricks/LTX-2 `fd4ded7f`: `ltx-pipelines/docs/multimodal-guidance.md:13` says "Set to `[]` to disable STG", in the same table and the same idiom as `stg_scale` -> 0.0 and `cfg_scale` -> 1.0; `MultiModalGuiderParams.stg_blocks` DEFAULTS to `[]` (`guiders.py:204`, `field(default_factory=list)`); `--audio-stg-blocks` is `nargs="*"` (`args.py:1039-1045`, `:1107-1113`) so the empty list has a CLI spelling, and `nargs="+"` was the one-character way to forbid it; `LTX_2_3_HQ_PARAMS` SHIPS `stg_blocks=[]` on both modalities (`constants.py:105,:113`); and a whole-tree search found NO validation of `stg_blocks` anywhere - no emptiness, length or range check, no assert, no raise - with the null results recorded. `blocks=None` means EVERY block and `blocks=[]` means NO block (`perturbations.py:26-33`), so the empty list is how a caller says the second thing, and both this tree's parsers go out of their way to keep PRESENT-and-empty distinct from ABSENT - then refuse the one configuration that distinction exists to express. Upstream does not skip the pass either: `do_perturbed_generation` reads `stg_scale` alone (`guiders.py:279-281`), so an empty list disables the STG SIGNAL and not the STG COST. FOUND while repairing the fresh review of [#1092](https://github.com/mudler/vllm.cpp/issues/1092), whose B3 finding is the same divergence on the VIDEO path; that half IS fixed in flow (`ltx2_video.cpp` `ApplyGuidanceOverrides`, `ltx2_denoisers.cpp` `check_reaches_a_block`). NOT fixed here because this half landed with [#1032](https://github.com/mudler/vllm.cpp/issues/1032) and moving it changes a landed row's gated behaviour and its case "a perturbed pass over NO block is refused", which is the normal row-spec-review path rather than an in-flow repair. Until then the two arms disagree about the same request. Listed under `## Owed` in [`ltx25-guided-video.md`](specs/ltx25-guided-video.md) | bug | | [#837](https://github.com/mudler/vllm.cpp/issues/837) | `ROCM-GEMMA4-GETBLAS-DUALSLOT` | ROCm: dual-slot GetBlas TLS + host lifetime so two HIP streams do not share one handle | bug | +| [#1117](https://github.com/mudler/vllm.cpp/issues/1117) | `LTX25-A2VID-RECIPE` | `A2VidPipelineTwoStage` (`a2vid_two_stage.py:53` @ `fd4ded7f`) has no recipe row, so `pipeline_kind = a2vid_two_stage` gets the generic table refusal (`src/vllm/model_executor/models/ltx2_pipeline.cpp:1328-1332`) naming the pair rather than the missing machinery. [#922](https://github.com/mudler/vllm.cpp/issues/922) is CLOSED and closed the audio CONDITIONING, not the recipe: a supplied take rides `distilled_two_stage`, which [`ltx25-a2v-audio-input.md`](specs/ltx25-a2v-audio-input.md):438-446 already records as a different trajectory. Measured at `daeff67f2`: `git grep -n '"a2vid' -- src include tests docs examples` returns TWO hits, both upstream anchors inside `Fail`-message assertions (`tests/vllm/multimodal/test_ltx2_video.cpp:4363,:4427`), against a control of 4 for `"one_stage"` in `include/` alone. Four differences from the recipe it rides, each read at the pin: stage 1 is CFG/STG/modality-guided and caller-configured (`:230-240`, fed from `utils/args.py:947-1006`, `--a2v-guidance-scale` defaulting to `video_guider.modality_scale` = 3.0 at `utils/constants.py:54,:64`) where `distilled_two_stage` fixes `allow_guidance_override = false`; stage 1's schedule is scheduler-derived (`:225-227`) against our fixed `DistilledSigmas()`; stage 1 is plain Euler (`:229-258` passes no `stepper`, `utils/blocks.py:526-527`) against our `kEulerAncestral` on 2.5; and the AUDIO guider is the DEFAULT positive-only one (`:237-239`, `ltx-core components/guiders.py:200-210`) rather than the params table's cfg-7.0 row. Two non-schedule facts that must not be guessed: `--audio-path` is `required=True` (`:312-317`), and the distilled LoRA rides stage 2 ALONE (`:114` against `:107`) with `--distilled-lora` `required=True` (`utils/args.py:1140-1153`). Unblocked by `Ltx2GuidedDenoise` landing at `daeff67f2` (#1092/#1102), which [`ltx25-guided-video.md`](specs/ltx25-guided-video.md) `## Owed` names this arm against. Spec [`ltx25-a2vid-recipe.md`](specs/ltx25-a2vid-recipe.md) | enhancement | +| [#1118](https://github.com/mudler/vllm.cpp/issues/1118) | `LTX25-A2VID-RECIPE` | LoRA adapters fuse ONCE, at load, into ONE weight set: `src/vllm/multimodal/ltx2_video.cpp:816-820` is the only `dit_options.loras.push_back` in the tree and it runs under `if (!lora_path.empty())`, so every phase of every recipe sees the same fused `im.dit.weights`. Three upstream pipelines build TWO `DiffusionStage`s from the same checkpoint with DIFFERENT adapter sets at `fd4ded7f`: `a2vid_two_stage.py:107` against `:114-119`, `ti2vid_two_stages.py:140` against `:151`, and `ti2vid_two_stages_hq.py:154,:165` at two separate strengths (`:92-101`). `distilled.py:131` builds ONE stage set, which is why `distilled_two_stage`, `dfr` and `retake` have never needed this and no gate has ever asked. Consequence for the arm landing with #1117: that row mirrors `--distilled-lora required=True` (`utils/args.py:1140-1153`) by refusing an `a2vid_two_stage` load with no `lora_path`, so the 3-step stage-2 schedule cannot run on a checkpoint with no distilled adapter — but it CANNOT mirror upstream leaving stage 1 WITHOUT that adapter, so stage 1's guided schedule runs against base + distilled LoRA where upstream runs it against the base alone. That divergence RENDERS, and the PIXELS it renders are not upstream's: it moves the trajectory, so the frames themselves differ, while the frame count, the shapes, the sample rate and the errors are all exactly what they were — nothing in the SHAPE of the result says anything is wrong, which is why it is filed rather than left in a comment. It is not undetectable, and saying so would be the more damaging error: the instrument that WOULD see it is a real-weights comparison against upstream's own render on the same checkpoint, take and seed, upstream's stage 1 on the base weights against ours on base + distilled. Two fix shapes, neither chosen: a second `Ltx2DitWeights` per adapter set (what upstream pays, two `from_checkpoint` calls at `a2vid_two_stage.py:103,:115`) or unfused runtime LoRA selectable per phase; the first doubles resident DiT weights and the second changes the GEMM path. Bounds #1093 and #921, which need the same seam. Listed under `## Owed` in [`ltx25-a2vid-recipe.md`](specs/ltx25-a2vid-recipe.md) | enhancement | diff --git a/.agents/specs/ltx25-a2vid-recipe.md b/.agents/specs/ltx25-a2vid-recipe.md new file mode 100644 index 000000000..e360677e8 --- /dev/null +++ b/.agents/specs/ltx25-a2vid-recipe.md @@ -0,0 +1,461 @@ +# LTX25-A2VID-RECIPE — the audio-to-video recipe, and the take it has to consume + +Row `LTX25-A2VID-RECIPE`, under the campaign [`ltx-2-5.md`](ltx-2-5.md). +Issue [#1117](https://github.com/mudler/vllm.cpp/issues/1117). +Base: `origin/main` @ `daeff67f2`. +Upstream: Lightricks `LTX-2` @ `fd4ded7f2d88d3da713abcdd4ad41ecc4a9314ca`, verified +with `git rev-parse HEAD` in `/home/mudler/_git/LTX-2` before any anchor below was +taken. Paths are relative to `packages/ltx-pipelines/src/ltx_pipelines/` and +`packages/ltx-core/src/ltx_core/`, as every other LTX-2.5 spec here uses them. + +vLLM implements nothing in this class and neither does vLLM-Omni, whose recipe +table stops at the two-stage distilled row. Lightricks is the reference, and it +is the only one. + +## 0. Honesty statement — what this row claims and what it does not + +It claims: `pipeline_kind = a2vid_two_stage` resolves a recipe that mirrors +`A2VidPipelineTwoStage.__call__` phase for phase; that the recipe is reachable +from `include/vllm.h` and from `ltx2-gen` on its own default configuration; that +an a2vid render without a driving take is refused rather than rendered +unconditioned; and that the gate can see whether the frozen take was CONSUMED by +the DiT rather than merely carried in the recipe. + +It does not claim a render on real weights. The GPU is out of bounds for this +row, `dgx.casa` is not answering ssh, and every number below is measured on this +tree's own reduced fixture. It does not claim an oracle-run comparison: no +LTX-2.5 checkpoint here has a recorded sha256 (#1048) and the pinned secondary +oracle for this class is `vllm-omni`, which is UNPINNED (#633) and carries no +LTX-2.5 recipe at all. Every value below is read off upstream SOURCE at +`fd4ded7f`. + +It does not claim upstream's adapter placement. See §4.4 and +[#1118](https://github.com/mudler/vllm.cpp/issues/1118). + +## 1. The gap, verified on this tree at `daeff67f2` + +`git grep -n '"a2vid' -- src include tests docs examples` returns TWO hits, both +upstream anchors inside `Fail`-message assertions +(`tests/vllm/multimodal/test_ltx2_video.cpp:4363`, `:4427`). The control is +`git grep -c '"one_stage"' -- src include`, which returns 4 in the header and 1 +in the recipe table. So the pipeline kind genuinely does not exist, and the grep +is not failing on the wrong term. + +Issue [#922](https://github.com/mudler/vllm.cpp/issues/922) is CLOSED and closed +the **conditioning**, not the recipe. What it left is written down in its own +spec ([`ltx25-a2v-audio-input.md`](ltx25-a2v-audio-input.md):438-446): "This row +ports the audio-conditioning mechanism and rides the checkpoint's own resolved +recipe", which in practice is `distilled_two_stage`, "and no claim is made that a +render here reproduces upstream's A2Vid output." + +The blocker that entry named is gone. `Ltx2GuidedDenoise` landed at `daeff67f2` +(#1092/#1102) with production callers in `src/vllm/multimodal/ltx2_video.cpp`, +and [`ltx25-guided-video.md`](ltx25-guided-video.md) `## Owed` records +`a2vid_two_stage` as "its own row; none is blocked on this seam any more". + +**Two anchors in the dispatch that sent this row were wrong and are corrected +here rather than propagated.** The a2v guidance default is NOT 0.0 and is not at +`a2vid_two_stage.py:318-323`; `:318-323` is `--audio-start-time`, whose default +is 0.0. `--a2v-guidance-scale` lives at `utils/args.py:986-995` — the +`parser.add_argument(` call opens at `:986`, the flag is named at `:987` and the +default is set at `:989` — and it defaults to +`video_guider.modality_scale`, which is **3.0** (`utils/constants.py:54`, +`:64`). And "audio conditioning built at `:53`, called at `:143`" names the class +statement and the `__call__` signature; the `AudioConditioner` is constructed at +`:96-102` and called at `:200`. + +## 2. The port map, phase by phase + +Upstream's two stages, and the `Ltx2PhaseRecipe` each becomes. + +| Field | stage 1 | stage 2 | Upstream | +|---|---|---|---| +| `name` | `stage_1` | `stage_2` | the pipeline's own attribute names, `:103` and `:115` | +| `spatial_downscale` | 2 | 1 | `width // 2, height // 2` at `:206-212`; full shape at `:264` | +| `sigmas` | empty (derived) | `Stage2DistilledSigmas()` | `self._scheduler.execute(steps=num_inference_steps)` at `:225-227`; `stage_2_sigmas=STAGE_2_DISTILLED_SIGMAS` at `:164` | +| `use_official_sigma_schedule` | true | false | as above | +| `noise_scale` | 1.0 | 0.909375 | `ModalitySpec.noise_scale` default (`utils/types.py:110`) since `:247-250` sets none; `stage_2_sigmas[0].item()` at `:288` | +| `input_transform` | `kInitial` | `kSpatialUpsample` | `self.upsampler(video_state.latent[:1])` at `:261` | +| `video_guidance` | the params table's video row | defaults | `MultiModalGuider(params=video_guider_params, negative_context=v_context_n)` at `:233-236`, fed from `utils/args.py:947-1006` (the six video-guider flags, `--video-cfg-guidance-scale` at `:948` through `--video-skip-step` at `:997`; the audio group starts at `:1007`); `SimpleDenoiser(v_context_p, a_context_p)` at `:278` | +| `audio_guidance` | defaults | defaults | `MultiModalGuider(params=MultiModalGuiderParams())` at `:237-239` | +| `allow_guidance_override` | true | false | the CLI passes six video guider fields at `:353-360`; stage 2 takes no guider at all | +| `stepper` | `kEuler` | `kEuler` | `:229-258` and `:277-297` pass no `stepper`, so `EulerDiffusionStep()` applies (`utils/blocks.py:526-527`) | + +**The audio guider on stage 1 is the DEFAULT one and not the params table's audio +row.** `MultiModalGuiderParams()` at `:237-239` is cfg 1.0 / stg 0.0 / rescale +0.0 / modality 1.0 / stg_blocks `[]` (`ltx-core components/guiders.py:200-210`), +which is `_POSITIVE_ONLY_GUIDER`. The params table's audio row carries cfg 7.0 +(`utils/constants.py:61`). Taking the table's row here — which is what +`OneStagePhase` does, correctly, for `ti2vid_one_stage.py:215-218` — would run an +unconditional audio forward on a stream that is frozen, and would spend a text +encode on a negative context the audio guider never asked for. It renders either +way. + +**Stage 1 is plain Euler and this is the one place a neighbouring recipe is +actively misleading.** `DistilledTwoStageRecipe` selects `kEulerAncestral` for +stage 1 on generation 2.5, and that comes from `distilled.py:76-84`, which +reaches a2vid through nothing: `a2vid_two_stage.py` never imports +`should_use_ancestral_sampler` and never passes a `stepper`. Selecting it here +because the stage is "the two-stage first stage" would be inferring a sampler +from a neighbour. + +Recipe-level fields: + +| Field | Value | Upstream | +|---|---|---| +| `height` / `width` | `params.stage_2_*` | `parser.set_defaults(height=params.stage_2_height, ...)` at `utils/args.py:1128` | +| `num_inference_steps` | `params.num_inference_steps` | `:152`, consumed at `:226` | +| `negative_prompt` | `kOmniNegativePrompt` on the `2` and `2.3` rows, `LightricksNegativePrompt()` on `2.4` and `2.5` | `--negative-prompt` default `DEFAULT_NEGATIVE_PROMPT` (`utils/args.py:937-946`), consumed at `:176-183`. Which string is a question of WHICH REFERENCE owns the row, per the header's "which upstream owns which value": vLLM-Omni supplies the pre-2.4 rows and Lightricks the 2.4 and 2.5 ones. `t2a_one_stage` splits the same way at the same four versions (its arm of `ResolveLtx2PipelineRecipe`), and these rows mirror it one for one | +| `allow_negative_prompt` | true | `:146` is a parameter and `:183` reads `ctx_n` into the video guider | +| `allow_request_sigmas` | true | stage 1's schedule IS `num_inference_steps` (`:226`); stage 2's is a constant and carries its own explicit `sigmas`, which the engine reads before the override branch | +| `fixed_num_inference_steps` | false | as above | +| `video_output_phase` / `audio_output_phase` | 1 | `:299` decodes `video_state.latent` after stage 2 | +| `allow_request_latents` | false | `:229-297` construct every `ModalitySpec` from pipeline state; no request latent surface exists | +| `requires_audio_input` | true | `--audio-path` `required=True` at `:312-317` | +| `requires_distilled_lora` | true | `--distilled-lora` `required=True` at `utils/args.py:1140-1153` | + +**Which versions.** All four this table KEYS — `2`, `2.3`, `2.4`, `2.5` — +mirroring the `t2a_one_stage` rows and for the same reason: +`A2VidPipelineTwoStage` takes whatever `resolve_cli_params()` read off the +checkpoint (`:311`), exactly as `T2AOneStagePipeline` does at +`t2a_one_stage.py:178-179`. There is no "which generations support A2V" question +upstream, and restricting the rows would be a local invention. This differs from +`distilled_two_stage`'s two rows, which are two rows because two DIFFERENT +references supply them. + +Four KEYS is not four params objects, and the earlier wording — "all four the +params table distinguishes" — claimed the second. `_PARAMS_SINCE_VERSION` +(`utils/constants.py:130-133`) carries exactly TWO rows, `(2,4)` and `(2,3)`, +with `LTX_2_PARAMS` as the fall-through at `:179`. So 2.5 does not have a params +row of its own: it is at or above `(2,4)` and resolves onto the 2.4 one. +`Ltx2DetectPipelineParams` (`ltx2_pipeline.cpp:947-956`) mirrors that shape +exactly and its own comment already says so — "this is what gives LTX-2.5 the 2.4 +params". The four keys exist because the RECIPE table refuses an unknown +`(kind, version)` by name rather than defaulting, not because upstream reads four +different parameter sets. + +## 3. What already exists and is reused unchanged + +Nothing in the denoise loop moves. The row is a recipe row plus two refusals. + +| Piece | Where | Reached by this row how | +|---|---|---| +| decode → encode → truncate the take | `Ltx2DecodeAudioWav` / `Ltx2EncodeAudioToLatent`, `ltx2_video.cpp:2497-2516` | the `audio_path` extra, now REQUIRED on this kind | +| the freeze, both halves | the zeroed denoise mask at `ltx2_video.cpp:2994` and the scalar `ain.sigma` at `:3414` | unchanged; §5.2 gates that this arm consumes it | +| the guided seam | `Ltx2GuidedDenoise`, `ltx2_video.cpp:3483` | stage 1's guider is the params table's video row, so all four passes run | +| the spatial upsample | `Ltx2PhaseInputTransform::kSpatialUpsample`, `ltx2_video.cpp:2739` | stage 2 | +| the caller's own waveform as the soundtrack | `ltx2_video.cpp:3853` | keyed on the take being present, which this kind now guarantees | +| the resolution guard's divisor | `max_spatial_downscale()` | stage 1's downscale of 2 makes it 64, which is `assert_resolution(is_two_stage=True)` (`:168`) | + +`phase.noise_scale` is applied to both streams at `ltx2_video.cpp:3244-3248` and +that is NOT a divergence from upstream's per-modality `noise_scale=0.0`: +`ApplyGaussianNoise` is masked, and a frozen stream's mask is all zeros, so the +audio latent stays at `clean` whatever the phase's scale is. Stage 2's 0.909375 +therefore cannot reach the frozen take. Checked rather than assumed — §5.2's +digest is taken after the noiser, on both phases. + +## 4. Design + +### 4.1 Two recipe flags, not two string compares + +`Ltx2PipelineRecipe` grows `requires_audio_input` and +`requires_distilled_lora`. Both are flags on the recipe for the reason +`audio_only` already gives in the header: the engine has to answer the question +in a place that is not the recipe table, and an `im.pipeline_kind == +"a2vid_two_stage"` test at each site is one chance per site to miss the next +recipe that needs it. `ti2vid_two_stages` (#1093) and `keyframe_interpolation` +(#1096) both come with `--distilled-lora required=True`, so the second flag has a +second user before it lands. + +### 4.2 The audio refusal + +`requires_audio_input` is checked in `Generate`, beside the `audio_path` read, +because `pipeline_kind` is a LOAD knob and `audio_path` is a per-generation +extra: the question is only decidable once a request exists. The message names +the extra, the upstream line that makes it required, and what happens without it +— an unconditioned render that returns a clip of the right size, frame count and +sample rate, which is the shape of defect this file keeps finding. + +### 4.3 The LoRA refusal + +`requires_distilled_lora` is checked at LOAD, where `lora_path` is read +(`ltx2_video.cpp:808-820`), because both are load-time. Upstream cannot run this +pipeline without the adapter; neither can this. Refusing there rather than at +generate time means a caller learns before paying for a 22B load. + +The refusal deliberately does not try to verify that the supplied adapter IS the +distilled one. Upstream does not either: `--distilled-lora` takes any path. + +### 4.4 What this cannot mirror, and why it is filed rather than commented + +`stage_2_loras = (*loras, *distilled_lora)` at `:114` puts the distilled adapter +on stage 2 ALONE; stage 1 gets `loras=tuple(loras)` at `:107`. This engine fuses +at load into ONE weight set — `ltx2_video.cpp:816-820` is the only +`dit_options.loras.push_back` in the tree — so the adapter reaches both phases. +Stage 1's guided schedule therefore runs against base + distilled LoRA where +upstream runs it against the base alone — 30 steps of it on 2.5, since +`LTX_2_3_PARAMS` sets `num_inference_steps=30` (`utils/constants.py:85`) and 2.4 +inherits it at `:124`, which is the row 2.5 resolves onto. + +That divergence RENDERS, and the PIXELS it renders are not upstream's. It changes +the trajectory, so the frames themselves differ; what it leaves untouched is the +frame count, the shapes, the sample rate and the errors — nothing in the SHAPE of +the result says anything is wrong. It is therefore +[#1118](https://github.com/mudler/vllm.cpp/issues/1118) and `## Owed` rather than +a comment. It bounds #1093 and #921 the same way. + +**Say what a gate could see, not that none could.** "Changes nothing a caller can +read" would be false and would be the more damaging kind of false, because it +implies no instrument could ever detect this. A caller reads pixels. The gate +that WOULD detect it is the real-weights comparison against upstream's own render +that §0 and `## Owed` record this row as not having: same checkpoint, same take, +same seed, upstream's stage 1 on the base weights against ours on base + +distilled. That comparison is owed, not impossible, and #1118 is the row that +owes it. + +Refusing the whole kind instead was considered and rejected: it would land the +recipe dead, and the rule that forbids dead code is not satisfied by a capability +nobody can reach. Rendering with NO adapter was also rejected — a 3-step +distilled refinement on a checkpoint that was never distilled is the +plausible-and-wrong shape [`ltx25-a2v-audio-input.md`](ltx25-a2v-audio-input.md) +`## Owed` warned this row about by name. + +### 4.5 What the implementation added that §4 did not foresee + +`Ltx2PhaseDenoiser { kGuided, kSimple }`, a new phase field naming which of +upstream's two denoiser classes (`utils/denoisers.py`) a phase constructs. It +exists because **`allow_guidance_override` cannot express a2vid's stage 2**, and +that only became visible once the recipe was written. + +That boolean answers "does this pipeline's CLI carry the guider flags at all". +`distilled.py` selects `default_2_stage_distilled_arg_parser` +(`utils/args.py:1188`; `:1187` is blank), which never adds them, so an override +there names a knob the pipeline has no surface for and the engine refuses it — +correctly, and that refusal is landed and gated. `a2vid_two_stage.py:311` selects +`default_2_stage_arg_parser` (`utils/args.py:1123`), which DOES carry them +(`utils/args.py:947-1006`), +and they reach stage 1's guider alone (`:233-236`) because stage 2 constructs +`SimpleDenoiser(v_context_p, a_context_p)` (`:278`) and takes no params at all. + +Neither value of the boolean says that. `false` refuses a request upstream +accepts. `true` applies the override to stage 2's positive-only params and +switches on a guidance pass upstream's stage 2 does not run — invisibly, since an +extra forward changes no output shape, frame count or sample rate. So a2vid's +stage 2 is `allow_guidance_override = true` **and** `kSimple`, and +`ApplyGuidanceOverrides` skips a `kSimple` phase AFTER the refusal check. The +order matters: every existing recipe that refuses is also `kSimple`, so testing +the skip first would silently turn three landed refusals into silent ignores. + +The field is populated on every phase in the table from the upstream line that +constructs the denoiser, and it is read in exactly one place. + +**How it is gated, and why the obvious gate is vacuous.** The claim is that an +override reaches stage 1 and not stage 2, and no trace field records what the +second phase did. The instrument is a pair of renders whose only difference is a +value that is ALREADY stage 1's own: `video_stg_scale = 1.0` is what this +recipe's stage 1 carries (`utils/constants.py:52`), so applying it there changes +nothing, while stage 2's own STG scale is 0.0 and applying it THERE adds a +perturbed forward per step. Equal artifact bytes therefore mean the override +stopped at stage 1. The case `REQUIRE`s both of those recipe values first, so a +table change that made the restated value differ from stage 1's turns the +comparison into a failure rather than into a tautology. + +## 5. Tests and gates + +Focused: `ctest --test-dir build -R 'ltx2' --output-on-failure`, and the whole +`test_ltx2_video` / `test_ltx2_pipeline` binaries run with NO `--test-case` +filter, because a truncated filter matched zero cases and printed `SUCCESS!` with +exit 0 on a sibling row and many case names here contain commas. The case and +assertion counts are asserted non-zero. + +Full: + +``` +cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DVLLM_CPP_CUDA=OFF +cmake --build build -j6 && ctest --test-dir build -j4 --output-on-failure +``` + +Reported with `CONFIGURE_EXIT`, `BUILD_EXIT`, the `: error:` count, `ctest -N`, +`CTEST_EXIT`, the pass/fail line, positive controls for `No space left` and +`BFD assertion`, plus load and free disk. Known-red cited by the issue naming the +test, never by family: `windows-msvc-*` (#584), `test_async_llm` (#294), +`test_engine_core_proc` (#1052), `test_serve_low_tools` (#428), +`test_cpu_x86_llamacpp_floor` (#618). + +### 5.1 RED first, through the production entry point + +The smallest failing test loads an engine with +`pipeline_kind = a2vid_two_stage` and renders. It fails first because +`ResolveLtx2PipelineRecipe` refuses the pair by name. That is the intended +reason, and it is quoted in the pull request body. + +### 5.2 The frozen take is CONSUMED, not carried + +A recipe-level assertion proves the value is set. What this row asserts is that +the DiT saw it, on THIS arm, at every phase: + +- `trace.audio_frozen`, derived from the denoise mask the loop uses and read + AFTER the noiser (`ltx2_video.cpp:3261`) — the position that a mutation + survived when it was read before; +- `trace.audio_sigma_max == 0.0`, the scalar `Modality.sigma` half of upstream's + `frozen` (`utils/types.py:104-106`), which the mask cannot reach; +- `trace.audio_latent_digest` equal to the digest of the SAME render's take + encoded once, and `audio_latent_absmax > 0` so a zeroed latent cannot pass a + digest comparison against another zeroed latent. + +Two controls, because one of them alone is passable by a constant: + +- **the same take at a different SEED gives a bit-identical audio latent.** It is + the encoded file, not a sample. A build that noised the audio stream, or that + generated it and let the take decorate the trace, moves this digest and moves + nothing a caller can see. +- **a different `audio_start_time` gives a different one.** Without this the first + control passes on any constant tensor. + +### 5.3 The guidance arms, per arm and in x0 space + +The a2vid stage-1 guider is the params table's video row — cfg 3.0, stg 1.0, +rescale 0.7, modality 3.0 — so all four passes run and the rescale branch is +live. The per-arm invariant `x0 == latent - sigma*velocity` is asserted for +`cond`, `uncond`, `perturbed` and `modality` off `Ltx2ConditioningTrace`'s +recorded pass tensors. In velocity space the residual is the whole sample and the +RED prints `|x0 - velocity| = 0` exactly. + +Non-vacuity is `REQUIRE`d: a zero step-0 sample makes the two candidate tensors +coincide. + +### 5.4 The mutations this gate must survive + +| # | Mutation | Must go RED at | +|---|---|---| +| M1 | the `a2vid_two_stage` dispatch row deleted (reachability) | the whole e2e case | +| M2 | stage 1's `spatial_downscale` set to 1 | the phase-shape assertions | +| M3 | stage 1 given `DistilledSigmas()` instead of the derived schedule | the schedule assertions | +| M4 | stage 1's `allow_guidance_override` set to false | the override case | +| M5 | stage 1's audio guider taken from the params table | the audio-guider assertion | +| M6 | stage 1's stepper set to `kEulerAncestral` | the stepper assertion | +| M7 | `requires_audio_input` never checked | the no-take refusal case | +| M8 | `requires_distilled_lora` never checked | the no-LoRA refusal case | +| M9 | the frozen take's denoise mask left at 1 on this arm | §5.2 `audio_frozen` | +| M10 | the frozen scalar sigma left at the schedule's | §5.2 `audio_sigma_max` | +| M11 | every guidance arm left in velocity space | §5.3, all four rows | +| M12 | the `kSimple` skip deleted, so an override reaches stage 2 | §4.5's artifact comparison | + +Each mutation reports three facts: `git diff --stat`, whether it BUILT with the +compile-error count, and the exit code captured directly. A mutation that fails +to compile, and a mutation that never applied, both read exactly like a passing +test. The mutations are run against the COMMITTED head, so that first fact is +the mutation's own diff rather than the whole uncommitted change. + +**Measured: twelve applied, twelve DETECTED, and one of them by one binary +only.** M6 — the stepper — is SURVIVED by the end-to-end case and DETECTED by the +recipe case (`test_ltx2_pipeline`, exit 1, 44 cases / 2598 assertions). That is +recorded as it measured rather than as it would read better: the stepper is a +recipe field, the end-to-end case has no baseline to compare a trajectory +against, and this recipe's `noise_seed_offset` is 0, so the ancestral arm moves +no digest the trace carries. + +**One harness defect, found and fixed in flow.** §4.5's artifact comparison was +written as `CHECK(a == b)` over PPM pixels and a WAV. A failing one dumps raw +bytes into the doctest report, and that killed the harness with a +`UnicodeDecodeError` between applying M12 and restoring it — the shape +[`ltx25-a2v-audio-input.md`](ltx25-a2v-audio-input.md) §5 already records. The +`finally` restored the tree, the comparison is now a differing-byte COUNT, and +the harness decodes with `errors="replace"`. + +## 5b. Reachability — the sentence the records must carry + +Entry point: `vllm_video_engine_load` → `LoadVideoEngine` with +`pipeline_kind = a2vid_two_stage` (a documented value of a documented load +extra), then `vllm_video_generate` → `VideoEngine::Generate` with the +`audio_path` extra. The chain is +`include/vllm.h` → `src/capi/vllm_c.cpp` → `Ltx2VideoEngine::Generate` → +`ResolveLtx2PipelineRecipe`'s a2vid row → the phase loop → `Ltx2GuidedDenoise`. +`ltx2-gen` is the same call through `--pipeline-kind` and `--audio-path` as a +thin ABI client that includes no internal header. + +The reachability mutation is M1: delete the `a2vid_two_stage` dispatch row so the +table refuses the pair, and rerun the focused gate. + +**What does NOT reach it, stated rather than left to be found.** The OpenAI +`/v1/videos` route cannot drive this: `VideoGenParamsFromRequest` +(`src/vllm/multimodal/video_engine.cpp:349-384`) never writes `gen.extras`, so no +per-generation extra reaches any engine over HTTP +([#928](https://github.com/mudler/vllm.cpp/issues/928)). `pipeline_kind` is a +LOAD extra and IS reachable over `--video-extra`, so a server can be started on +this kind — and every such request would then be refused for the missing take, +which is the correct behaviour and not a way to use the route. The honest +statement is: reachable from `include/vllm.h` and `ltx2-gen`; NOT drivable over +`/v1/videos` until #928 is fixed. + +## 6. Quantized arms + +This row adds no GEMM, no kernel and no dtype. Arm by arm so none is left to be +discovered: + +| Arm | Disposition | +|---|---| +| bf16 / f32 safetensors | ported; the shipped audio VAE and the fixture are this | +| NVFP4, FP8 on the DiT tower | unaffected. A recipe row selects sigmas, guiders and phase shapes; both arms reach it exactly as they reach `distilled_two_stage` | +| GGUF k-quants | not applicable and not merely undone. Upstream enumerates its inference quantization kinds exhaustively — fp8-cast, fp8-scaled-mm, nvfp4-cast, nvfp4-prequant (`utils/quantization_factory.py:23-26`, `assert_never` at `:50`) — so there is no upstream GGUF behaviour to mirror, and llama.cpp does not carry this architecture, so there is no quant-matched comparison to serve | +| int8-convrot | out of scope, already refused by name at `ltx2_pipeline.cpp`'s `kInt8ConvRot` | +| the device-resident arm | REFUSED by name for this kind's default guidance, and that refusal already exists: stage 1's `stg_scale = 1.0` and `modality_scale = 3.0` both need a perturbed forward, and `Ltx2DitForwardDevice` takes no `perturbations` (`ltx2_video.cpp:2598-2618`). Unchanged by this row and named here so it is not discovered later | + +## 7. Risks + +**R1 — a recipe that renders whatever it says.** Every field below is a number +that produces a finished clip whether it is right or wrong. Mitigated by +asserting the recipe's fields directly against the upstream anchors in §2, and by +mutations M2 to M6, which each move exactly one of them. + +**R2 — the `READER ANCHORS` gate.** `ltx2_video.cpp:304-305` carries a +line-number list re-derived and string-compared by a case in +`test_ltx2_video.cpp`. Any line inserted above it shifts the list and a clean +`git merge` will not warn. Mitigated by re-deriving after the final merge of +`origin/main` and naming it as a merge hazard in the pull request body. + +**R3 — concurrent edits.** `ltx2_video.cpp`, `ltx2_pipeline.{h,cpp}`, +`docs/FEATURES.md` and `.agents/issue-index.md` are edited by sibling rows. +`docs/BENCHMARKS.md` and `docs/FEATURES.md` sit at their prose-paragraph budgets +(#1055), so this row puts its content in TABLE ROWS and runs +`check-public-doc-tables.py` before pushing. The issue index is taken from +`origin/main` wholesale and re-appended on any conflict, never auto-merged. + +**R4 — the adapter divergence in §4.4.** Named, filed as #1118, and listed under +`## Owed`. + +## 8. Stop conditions + +Report `NEEDS_DECISION` rather than narrowing silently if: + +- the fixture cannot supply a LoRA, because then `requires_distilled_lora` makes + the kind unreachable in-tree and the row becomes seam-only; +- stage 1's derived schedule cannot be exercised on the fixture without the + guided seam refusing (the fixture DiT has two blocks and the params row names + block 28 — `LTX_2_3_PARAMS` overrides 2.0's `[29]` to `[28]` at + `utils/constants.py:86`, and 2.4, which 2.5 resolves onto, inherits it at + `:124`), because then the guided arm of this recipe is gated by nothing. + +## Owed + +- **[#1118](https://github.com/mudler/vllm.cpp/issues/1118) — the per-phase + adapter.** §4.4. Owned by this row. It also bounds + [#1093](https://github.com/mudler/vllm.cpp/issues/1093) and + [#921](https://github.com/mudler/vllm.cpp/issues/921). +- **A real-checkpoint A2V render.** Gated on fixtures only. The artifacts exist + now — `/usr/local/nas_share/checkpoints/ltx-2.5/lightricks-ltx-2.5/` holds the + NVFP4 DiT, both video VAEs, the audio VAE, both upscalers and + `loras/ltx-2.5-22b-distilled-lora-450-bf16.safetensors` — and the host does + not: `dgx.casa` is network-alive and its sshd will not handshake. +- **An oracle-run comparison.** `vllm-omni` is UNPINNED (#633) and carries no + LTX-2.5 recipe; no LTX-2.5 checkpoint here has a recorded sha256 (#1048). The + recipe is gated against upstream SOURCE, not upstream OUTPUT, and that is the + ceiling on this row's evidence. +- **`/v1/videos` cannot drive this** until + [#928](https://github.com/mudler/vllm.cpp/issues/928) forwards per-generation + extras. §5b. + +## Now + +`ACTIVE` — spec committed before implementation, per `AGENTS.md` § *Spec before +code*. diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 8cdf9c5f8..70a1b5f78 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -178,6 +178,7 @@ in `ltx2_text_encoder.cpp` is the call that would have to change. | LTX-2.5 T2A guidance space | LTX-2.5 DiT (T2A arm) | `test_ltx2_video` "the guider is handed x0 predictions" through `Generate`, on all 3 arms plus the guider output and the Euler input; a seam case puts the two spaces 1.5e-07 apart at rescale 0 and 0.352 at 0.7 | Combines **denoised (x0)**, mirroring `X0Model` (`model.py:590-604`). Was velocity space, which agrees only at rescale 0 ([#1039](https://github.com/mudler/vllm.cpp/issues/1039)) | | LTX-2.5 VIDEO guidance | LTX-2.5 DiT, joint video+audio | `test_ltx2_video`'s `ltx2 one_stage:` cases through `Generate`; all FOUR arms carry the x0 invariant and the guider output replays EXACTLY | `--pipeline-kind one_stage` runs `_guided_denoise`: 4 forwards/step, combined per modality in **x0**. Was ONE unguided forward, every `video_guidance` field dead ([#1092](https://github.com/mudler/vllm.cpp/issues/1092)) | | LTX-2.5 cross-attention perturbations | LTX-2.5 DiT | `test_ltx2_video` gates each direction ALONE, on a forward where one stream is PRESENT but DISABLED so only that one runs: the flag moves the stream it writes, the other leaves it bit-identical. Swapping the two is RED | `SKIP_A2V_CROSS_ATTN` / `SKIP_V2A_CROSS_ATTN` ported, which `modality_scale = 3.0` selects on every video row. The DEVICE forward takes no perturbations, so that pass is refused there by name | +| LTX-2.5 audio-to-video (`A2VidPipelineTwoStage`) | LTX-2.5 DiT + audio VAE encoder + spatial upsampler | `test_ltx2_pipeline` and `test_ltx2_video`'s `ltx2 a2vid:` cases through `LoadVideoEngine`+`Generate`; the take's latent is bit-identical across SEEDS and moves with the WINDOW | `--pipeline-kind a2vid_two_stage`. Guided half-res stage 1, DERIVED schedule, plain Euler; distilled 3-sigma stage 2. `--audio-path` and `--lora` REQUIRED; adapter placement diverges (#1118) | | LTX-2.5 guidance knobs | LTX-2.5 request surface | `test_ltx2_video` renders with an override and refuses one on a fixed recipe | Seven video/audio guider extras mirroring `default_1_stage_arg_parser`, plus a negative embeds pair for a tower-less engine. Refused whole on `distilled_two_stage` and `retake`, whose guidance is distilled in | | MTP speculator | Qwen3.6-27B, Qwen3.6-35B-A3B | token-identical to vLLM `mtp` at c1 | ~4% faster c1; +16% output tput (MoE) | | MTP speculation DEPTH (`num_speculative_tokens` > 1) | Qwen3.5/3.6 `mtp.*` heads | k=1..4 through the loader, greedy tokens unmoved, two witnesses per arm: the draft decode forwards the propose RAN, and whether the DELIVERED draft row varied with depth. `test_mtp_depth` 5/5, 63 assertions | Default stays k=1. NO speed claim at k>1. Drafts are proposed and verified, never ACCEPTED, and neither witness proves per-column provenance. Both await the owed DGX gate (#81) | diff --git a/docs/USAGE.md b/docs/USAGE.md index 57d0b8beb..3816f41b7 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -3193,25 +3193,28 @@ CHECKPOINT_ROOT=... VLLM_CPP_LTX2_TOWER_E2E=1 \ Recipes resolve on an EXACT `(pipeline_kind, model_version)` pair and refuse anything else by name rather than defaulting, because a plausible but wrong sigma -schedule or guidance scale renders a video instead of failing. **Sixteen** pairs +schedule or guidance scale renders a video instead of failing. **Twenty** pairs resolve, derived from `ResolveLtx2PipelineRecipe`: -| `pipeline_kind` | resolving `model_version` | -|---|---| -| `one_stage` | 2, 2.3, 2.4, 2.5 | -| `distilled_two_stage` | 2, 2.5 | -| `res2s_two_stage` | **2.5 only** | -| `dfr` | **2.5 only** | -| `dmd2` | 2, 2.3 | -| `retake` | 2, 2.5 | -| `t2a_one_stage` | 2, 2.3, 2.4, 2.5 | +| `pipeline_kind` | resolving `model_version` | what it also needs | +|---|---|---| +| `one_stage` | 2, 2.3, 2.4, 2.5 | — | +| `distilled_two_stage` | 2, 2.5 | `upsampler_path` for its second phase | +| `res2s_two_stage` | **2.5 only** | `upsampler_path` for its second phase | +| `dfr` | **2.5 only** | `upsampler_path` | +| `dmd2` | 2, 2.3 | — | +| `retake` | 2, 2.5 | a source clip as a `frame_%06d.ppm` directory | +| `t2a_one_stage` | 2, 2.3, 2.4, 2.5 | a text tower; no video VAE is asked for | +| `a2vid_two_stage` | 2, 2.3, 2.4, 2.5 | `upsampler_path`, `lora_path`, and an `audio_path` on every request | This list ran to ten until 2026-08-17, omitting `dfr` entirely and all four `t2a_one_stage` rows. **`dfr` at 2 is refused deliberately, not by oversight**: DFR's base stage rests on generated keyframe slots, which need a checkpoint declaring `use_keyframes_abs_pos_embedding`, and the 2.0 distilled row predates that parameter — so resolving DFR onto it would build a recipe the engine must -then refuse at load. Refusing at the recipe table names the version instead. +then refuse at load. Refusing at the recipe table names the version instead +(the `dfr` arm of `ResolveLtx2PipelineRecipe`, named rather than given as a line +range because this row's own insertions above it staled the range once already). ### `res2s_two_stage`: the high-quality preset, and why it is a sampler @@ -3260,6 +3263,63 @@ ships with. And stage 1's guidance asks for an isolated-modality pass, which the device-resident forward cannot perturb, so this preset is host-only until that is closed; both are recorded in `.agents/specs/ltx25-res2s-loop.md`. +### Audio-to-video: rendering a clip around a soundtrack you supply + +`a2vid_two_stage` is `A2VidPipelineTwoStage`. Stage 1 denoises video at half +resolution, guided, on a schedule derived from the recipe's own step count; +stage 2 upsamples 2x and refines with the distilled three-sigma schedule. The +soundtrack is your file throughout: it is encoded once, frozen at both stages, +and handed back unchanged rather than round-tripped through the VAE. + +```sh +ltx2-gen --dit ltx-2.5-22b-distilled-fp8.safetensors \ + --dit-config ltx-2.5-transformer-config.json \ + --video-vae ltx-2.5-video-vae-conv-bf16.safetensors \ + --audio-vae ltx-2.5-audio-vae-bf16.safetensors \ + --upsampler ltx-2.5-latent-spatial-upscaler-x2-bf16-1.0.safetensors \ + --lora ltx-2.5-22b-distilled-lora-450-bf16.safetensors \ + --pipeline-kind a2vid_two_stage --audio-path take.wav \ + --prompt "a drummer in a small club" \ + --width 128 --height 128 --frames 25 --out out/a2v +``` + +**No render on real weights is claimed for this recipe.** It is gated on reduced +fixtures. Upstream's stage 1 runs the base `-dev-` transformer and puts the +distilled adapter on stage 2 only; the command above names the distilled +checkpoint this tree has measured elsewhere, so it is a shape to copy rather than +a reproduced result. + +Three things this kind demands, each refused by name rather than defaulted: + +| What | Why | Where upstream says so | +|---|---|---| +| `--audio-path` on **every** request | the pipeline is "denoise video around this take"; without one the soundtrack is generated and the clip looks finished | `--audio-path` is `required=True`, `a2vid_two_stage.py:312-317` | +| `--lora` naming the distilled adapter | stage 2 is a three-sigma refinement the base weights were never distilled for | `--distilled-lora` is `required=True`, `utils/args.py:1140-1153` | +| `--upsampler` | stage 2's input is the upsampled stage-1 latent | `a2vid_two_stage.py:261` | + +`--audio-start-time` and `--audio-max-duration` window the take; the window +defaults to the clip's own duration. A take shorter than the clip is refused +rather than padded, and a longer one keeps its leading frames. + +**One divergence, and it is not repairable from the request.** Upstream fuses +the distilled adapter into stage 2 alone and leaves stage 1 on the base weights; +this engine fuses adapters once at load, so stage 1 sees it too. Expect frames +that differ from the ones upstream renders for the same checkpoint, take and +seed. Nothing in the shape of the output shows it — the clip comes back at the +size, frame count and sample rate you asked for, and no error is raised — so the +only instrument that sees this is a side-by-side render against upstream. +Tracked as [#1118](https://github.com/mudler/vllm.cpp/issues/1118). + +The guider flags (`--video-cfg-guidance-scale` and the rest, spelled as the +`video_cfg_guidance_scale` extras over the C API) reach stage 1 and are ignored +by stage 2, which runs no guider at all — unlike `distilled_two_stage` and +`retake`, which refuse them outright. `pipeline_kind` is a LOAD knob and reaches +a server through `--video-extra pipeline_kind=a2vid_two_stage`, but `audio_path` +is a per-generation extra and `/v1/videos` forwards none +([#928](https://github.com/mudler/vllm.cpp/issues/928)), so every request to such +a server is refused for the missing take. This kind is reachable from the C API +and from `ltx2-gen`, and not over HTTP. + ### Retake: regenerating a time window of an existing clip `retake` is `RetakePipeline`: it keeps the source clip outside a window and diff --git a/examples/ltx2_gen/main.cpp b/examples/ltx2_gen/main.cpp index f94908d04..6da519d2b 100644 --- a/examples/ltx2_gen/main.cpp +++ b/examples/ltx2_gen/main.cpp @@ -195,7 +195,21 @@ const char* Need(int argc, char** argv, int i, const char* flag) { "--negative-prompt-embeds with --negative-audio-prompt-embeds. Absent both, a\n" "cfg scale other than 1.0 is refused by name. --pipeline-kind\n" "distilled_two_stage and retake distil their guidance INTO the weights and\n" - "refuse every one of these flags rather than applying it.\n"); + "refuse every one of these flags rather than applying it.\n\n" + "AUDIO-TO-VIDEO renders a clip AROUND a soundtrack you supply.\n" + "--pipeline-kind a2vid_two_stage selects it: stage 1 denoises video at half\n" + "resolution, guided by the checkpoint generation's own scales on a schedule\n" + "derived from the step count, and stage 2 upsamples 2x and refines with the\n" + "distilled three-sigma schedule. The take is encoded once and frozen at both\n" + "stages, and the audio.wav you get back is your own file. --audio-path is\n" + "REQUIRED on every request and --lora is REQUIRED at load, because upstream\n" + "makes both required and stage 2 is a refinement the base weights were never\n" + "distilled for; --upsampler is needed for stage 2 as on any two-stage recipe.\n" + "The guidance flags above reach stage 1 and are IGNORED by stage 2, which runs\n" + "no guider at all -- unlike distilled_two_stage, which refuses them. ONE\n" + "divergence you cannot fix from the command line: upstream puts the distilled\n" + "adapter on stage 2 alone and leaves stage 1 on the base weights, and this\n" + "engine fuses adapters once at load, so stage 1 sees it too (issue 1118).\n"); std::exit(code); } diff --git a/include/vllm/model_executor/models/ltx2_pipeline.h b/include/vllm/model_executor/models/ltx2_pipeline.h index 78bc0ad42..c611af23d 100644 --- a/include/vllm/model_executor/models/ltx2_pipeline.h +++ b/include/vllm/model_executor/models/ltx2_pipeline.h @@ -573,6 +573,35 @@ enum class Ltx2PhaseInputTransform { kInitial, kSpatialUpsample }; // Row LTX25-RES2S-LOOP, issue #921. Spec .agents/specs/ltx25-res2s-loop.md. enum class Ltx2StepperKind { kEuler, kEulerAncestral, kRes2s }; +// Which denoiser upstream CONSTRUCTS for this phase — the two classes in +// ltx-pipelines `utils/denoisers.py`. `kGuided` is `GuidedDenoiser`, built from +// a `MultiModalGuider` per stream; `kSimple` is `SimpleDenoiser`, "single +// transformer call, no guidance" (`utils/denoisers.py:3`). +// +// THIS DOES NOT GATE THE SEAM, and reading it as if it did is the mistake worth +// naming here. `Ltx2GuidedDenoise` runs on EVERY phase, because a phase whose +// recipe sets no guidance keeps `Ltx2MultiModalGuiderParams`'s own defaults and +// those ARE `_POSITIVE_ONLY_GUIDER` (denoisers.py:25-28) — one pass, and a +// `calculate` whose every term is zero, which is `SimpleDenoiser`'s output. +// That equivalence is measured rather than argued; see +// .agents/specs/ltx25-guided-video.md section 10. +// +// What it DOES decide is where a request's guider override lands, and it exists +// because `allow_guidance_override` alone cannot express the a2vid case. That +// field answers "does this pipeline's CLI carry the guider flags at all": +// `distilled.py` selects `default_2_stage_distilled_arg_parser` +// (utils/args.py:1188), which never adds them, so an override there names a knob +// the pipeline has no surface for and is REFUSED. `a2vid_two_stage.py:311` +// selects `default_2_stage_arg_parser` (utils/args.py:1123), which DOES carry +// them (utils/args.py:947-1006, the six video-guider flags) — and they reach +// stage 1's guider alone (`:233-236`), +// because stage 2 constructs `SimpleDenoiser(v_context_p, a_context_p)` +// (`:278`) and takes no params at all. So on that phase the flag is legal and +// simply does not arrive. Neither value of a boolean says that: refusing would +// reject a request upstream accepts, and applying would switch on guidance +// upstream's stage 2 does not have. +enum class Ltx2PhaseDenoiser { kGuided, kSimple }; + // LTXPhaseRecipe (ltx2_recipes.py:29-50). struct Ltx2PhaseRecipe { std::string name; @@ -585,6 +614,10 @@ struct Ltx2PhaseRecipe { double noise_scale = 0.0; Ltx2PhaseInputTransform input_transform = Ltx2PhaseInputTransform::kInitial; bool allow_guidance_override = true; + // See `Ltx2PhaseDenoiser`. Read in exactly one place — where a request's + // guider overrides are applied — and only AFTER the refusal above, so no + // recipe that refuses an override can reach it. + Ltx2PhaseDenoiser denoiser = Ltx2PhaseDenoiser::kGuided; bool use_official_sigma_schedule = true; Ltx2StepperKind stepper = Ltx2StepperKind::kEuler; double stepper_eta = 0.0; @@ -626,6 +659,38 @@ struct Ltx2PipelineRecipe { // next audio-only recipe. The recipe table is the one place that knows. bool audio_only = false; + // `A2VidPipelineTwoStage` (a2vid_two_stage.py:53). TRUE means a driving + // waveform is not optional: `--audio-path` is `required=True` (`:312-317`) and + // the whole pipeline is "denoise video AROUND this take", with the audio + // stream frozen at both stages (`:251-256`, `:291-296`). + // + // FLAGS ON THE RECIPE, not `pipeline_kind` string compares at the two call + // sites, for the reason `audio_only` gives above. The second one already has a + // second user waiting: `ti2vid_two_stages` (#1093) and + // `keyframe_interpolation` (#1096) both select a parser where + // `--distilled-lora` is `required=True` (utils/args.py:1140-1153). + // + // WITHOUT THE TAKE the render still finishes. It returns a clip of the right + // size, the right frame count and the right sample rate, with the soundtrack + // generated rather than supplied — which is the ordinary joint-generation + // behaviour and is indistinguishable from audio-to-video that ignored its + // input. + bool requires_audio_input = false; + // `--distilled-lora` is `required=True` on the two-stage parser this pipeline + // selects (utils/args.py:1140-1153, `default_2_stage_arg_parser` at `:1123`), + // and stage 2's three-sigma refinement (`:164`) is what that adapter was + // trained for. A recipe that fixes this flag cannot render on a checkpoint + // carrying no adapter without running a distilled schedule on undistilled + // weights. + // + // What this flag CANNOT express is upstream's placement: + // `stage_2_loras = (*loras, *distilled_lora)` (a2vid_two_stage.py:114) puts + // the adapter on stage 2 ALONE, against `loras=tuple(loras)` for stage 1 + // (`:107`), and this engine fuses at load into one weight set. Owed by + // https://github.com/mudler/vllm.cpp/issues/1118 and recorded in + // .agents/specs/ltx25-a2vid-recipe.md section 4.4. + bool requires_distilled_lora = false; + int64_t max_spatial_downscale() const; }; @@ -704,6 +769,21 @@ void Ltx2AssertResolution(int64_t height, int64_t width, int64_t divisor); // `use_keyframes_abs_pos_embedding` // ("retake", "2") Lightricks retake.py:85,287,290-294,313-324 // ("retake", "2.5") same +// ("a2vid_two_stage", "2") Lightricks a2vid_two_stage.py:53,143 (row +// ("a2vid_two_stage", "2.3") LTX25-A2VID-RECIPE, #1117). Stage 1 denoises +// ("a2vid_two_stage", "2.4") VIDEO at half resolution, guided by the +// ("a2vid_two_stage", "2.5") params table's video row and a scheduler- +// DERIVED schedule (:225-227), with the audio +// stream frozen on the caller's own take +// (:251-256); stage 2 upsamples 2x and refines +// with STAGE_2_DISTILLED_SIGMAS and no guider +// at all (:277-297). It is NOT +// `distilled_two_stage` with a take attached: +// that recipe fixes both stages' sigmas, fixes +// its guidance, and samples stage 1 with the +// ANCESTRAL stepper on 2.5, where A2Vid passes +// no `stepper` and gets `EulerDiffusionStep()` +// (utils/blocks.py:526-527) // ("t2a_one_stage", "2") Lightricks t2a_one_stage.py:43,109 (row // ("t2a_one_stage", "2.3") LTX25-T2A-ONE-STAGE, #1005). The one_stage // ("t2a_one_stage", "2.4") rows' own schedule with `audio_only` set: diff --git a/src/vllm/model_executor/models/ltx2_pipeline.cpp b/src/vllm/model_executor/models/ltx2_pipeline.cpp index ba13d4b06..ddf8772a4 100644 --- a/src/vllm/model_executor/models/ltx2_pipeline.cpp +++ b/src/vllm/model_executor/models/ltx2_pipeline.cpp @@ -1221,6 +1221,11 @@ Ltx2PipelineRecipe DistilledTwoStageRecipe(const std::string& version) { stage1.spatial_downscale = 2; stage1.sigmas = DistilledSigmas(); stage1.noise_scale = 1.0; + // `SimpleDenoiser` on BOTH stages (distilled.py:266, :295). Recorded for the + // reader rather than read by anything on this recipe: the refusal below fires + // first, because `default_2_stage_distilled_arg_parser` (utils/args.py:1188) + // never adds the guider flags in the first place. + stage1.denoiser = Ltx2PhaseDenoiser::kSimple; stage1.allow_guidance_override = false; stage1.use_official_sigma_schedule = false; if (Ltx2ShouldUseAncestralSampler(version)) { @@ -1237,6 +1242,7 @@ Ltx2PipelineRecipe DistilledTwoStageRecipe(const std::string& version) { // sigma, which is what makes the upsampled latent valid at that noise level. stage2.noise_scale = Stage2DistilledSigmas().front(); stage2.input_transform = Ltx2PhaseInputTransform::kSpatialUpsample; + stage2.denoiser = Ltx2PhaseDenoiser::kSimple; // distilled.py:295 stage2.allow_guidance_override = false; stage2.use_official_sigma_schedule = false; // Always deterministic: a 3-step refinement cannot remove freshly injected @@ -1307,6 +1313,7 @@ Ltx2PipelineRecipe RetakeRecipe(const std::string& version) { stage.spatial_downscale = 1; stage.sigmas = DistilledSigmas(); stage.noise_scale = 1.0; + stage.denoiser = Ltx2PhaseDenoiser::kSimple; // retake.py:291-294 stage.allow_guidance_override = false; stage.use_official_sigma_schedule = false; // NOT the ancestral sampler. `Ltx2ShouldUseAncestralSampler` is @@ -1469,6 +1476,127 @@ Ltx2PipelineRecipe Res2sTwoStageRecipe(const std::string& version) { return recipe; } +// `A2VidPipelineTwoStage` (a2vid_two_stage.py:53). Row LTX25-A2VID-RECIPE, +// issue #1117. +// +// TWO stages that share only their frame count. Stage 1 is the FULL model, +// guided, at half resolution, on a schedule derived from `num_inference_steps` +// (`:225-227`); stage 2 is the spatial upsample plus a three-sigma distilled +// refinement with NO guider (`:277-297`). The audio stream is the caller's +// encoded take, frozen at both stages with `noise_scale=0.0` (`:251-256`, +// `:291-296`), and the soundtrack handed back is the caller's own waveform +// rather than a VAE round trip of it (`:301-303`). +// +// WHY THIS IS NOT `DistilledTwoStageRecipe` WITH A TAKE ATTACHED, which is what +// a supplied `audio_path` rides today. Four fields differ and each of them +// renders: +// +// * stage 1's SIGMAS are derived, not the frozen `DistilledSigmas()`; +// * stage 1's GUIDANCE is the params table's video row and is caller- +// overridable (`:353-360` passes six fields), against `allow_guidance_ +// override = false` on both distilled phases; +// * stage 1's STEPPER is plain Euler — `:229-258` passes no `stepper`, so +// `EulerDiffusionStep()` applies (utils/blocks.py:526-527) — against the +// ancestral one `distilled.py:76-84` selects for generation 2.5; +// * the AUDIO guider is the DEFAULT `MultiModalGuiderParams()` (`:237-239`), +// not the params table's audio row. +// +// THE AUDIO GUIDER IS THE ONE A READER IS MOST LIKELY TO "FIX". `OneStagePhase` +// takes the table's audio row and is right to: `ti2vid_one_stage.py:215-218` +// builds it from `audio_guider_params`. A2Vid does not, and cannot sensibly — +// the stream it would guide is frozen — so its cfg 7.0 would buy an +// unconditional forward and a negative text encode for a delta multiplied into a +// latent the sampler cannot move. +Ltx2PipelineRecipe A2VidTwoStageRecipe(const Ltx2PipelineParams& params, + const std::string& negative_prompt) { + Ltx2PipelineRecipe recipe; + + Ltx2PhaseRecipe stage1; + // Upstream's own attribute names (`:103`, `:115`). These are what a refusal + // quotes back to a caller, and "generate_lowres" would name the distilled + // recipe's phase rather than this one. + stage1.name = "stage_1"; + // `width // 2, height // 2` (`:206-212`). This is also what makes + // `assert_resolution(is_two_stage=True)` (`:168`) the 64-divisor arm here: + // `max_spatial_downscale()` derives the divisor from this field. + stage1.spatial_downscale = 2; + // EMPTY on purpose: `self._scheduler.execute(steps=num_inference_steps)` + // (`:225-227`) is resolved at run time from the request's step count, exactly + // as the `one_stage` rows are. + stage1.sigmas = {}; + stage1.use_official_sigma_schedule = true; + // `ModalitySpec.noise_scale` defaults to 1.0 (utils/types.py:110) and + // `:247-250` sets none, so stage 1 starts from pure noise. #1013 is the defect + // this line exists to not repeat: at 0.0 the state stays as + // `create_initial_state` wrote it, which with no initial latent is all zeros, + // and a zero-initialized denoise still returns a finite clip. + stage1.noise_scale = 1.0; + stage1.video_guidance = params.video_guider; + // `MultiModalGuiderParams()` — the default-constructed positive-only guider + // (`:237-239`, ltx-core components/guiders.py:200-210). Left at the struct's + // own defaults rather than zeroed field by field, so the two spellings of + // "no guidance" cannot drift. + stage1.audio_guidance = Ltx2MultiModalGuiderParams(); + // `GuidedDenoiser(...)` at `:230-240`, and the CLI passes six of its fields + // per request at `:353-360`. + stage1.denoiser = Ltx2PhaseDenoiser::kGuided; + stage1.allow_guidance_override = true; + stage1.stepper = Ltx2StepperKind::kEuler; + + Ltx2PhaseRecipe stage2; + stage2.name = "stage_2"; + stage2.spatial_downscale = 1; + // `stage_2_sigmas: torch.Tensor = STAGE_2_DISTILLED_SIGMAS` (`:164`). + stage2.sigmas = Stage2DistilledSigmas(); + stage2.use_official_sigma_schedule = false; + // `noise_scale=stage_2_sigmas[0].item()` (`:288`) — the upsampled latent is + // only valid at the noise level this stage starts from. + stage2.noise_scale = Stage2DistilledSigmas().front(); + stage2.input_transform = Ltx2PhaseInputTransform::kSpatialUpsample; + // `SimpleDenoiser(v_context_p, a_context_p)` (`:278`) takes no params at all, + // so both guider fields stay at the positive-only defaults. + stage2.denoiser = Ltx2PhaseDenoiser::kSimple; + // TRUE, and this is the pair `allow_guidance_override` alone cannot express. + // The flags DO exist on this pipeline's parser (`:311` selects + // `default_2_stage_arg_parser`), so a request carrying one is legal — it just + // reaches stage 1's guider and nothing else (`:233-236`). Refusing it here + // would reject a request upstream accepts; applying it would switch on + // guidance that upstream's stage 2 does not have. `kSimple` above is what + // makes the second half true. + stage2.allow_guidance_override = true; + stage2.stepper = Ltx2StepperKind::kEuler; + + recipe.phases = {stage1, stage2}; + // `default_2_stage_arg_parser` sets the request geometry to the FINAL output + // (utils/args.py:1128); stage 1 runs at half through `spatial_downscale`. + recipe.height = params.stage_2_height(); + recipe.width = params.stage_2_width(); + recipe.num_frames = params.num_frames; + recipe.frame_rate = params.frame_rate; + recipe.num_inference_steps = params.num_inference_steps; + recipe.default_image_crf = params.default_image_crf; + recipe.negative_prompt = negative_prompt; + recipe.video_output_phase = 1; + recipe.audio_output_phase = 1; + // Stage 1's schedule IS the step count (`:226`), so a `steps` override is + // upstream's `--num-inference-steps` and is honoured. Stage 2 carries its own + // explicit `sigmas`, which the engine reads before it consults this flag, so + // the override cannot reach the distilled refinement. + recipe.allow_request_sigmas = true; + recipe.fixed_num_inference_steps = false; + // `:229-297` build every `ModalitySpec` from pipeline state — stage 1's video + // spec carries no `initial_latent` at all, and stage 2's is the upsampler's + // output. There is no request-latent surface to honour. + recipe.allow_request_latents = false; + // `:146` takes a negative prompt and `:183` reads `ctx_n` into the video + // guider's `negative_context`, so unlike the distilled rows this one has a + // second encode to do whenever `cfg_scale != 1.0`. + recipe.allow_negative_prompt = true; + recipe.requires_audio_input = true; + recipe.requires_distilled_lora = true; + return recipe; +} + } // namespace Ltx2PipelineRecipe ResolveLtx2PipelineRecipe(const std::string& pipeline_kind, @@ -1506,6 +1634,30 @@ Ltx2PipelineRecipe ResolveLtx2PipelineRecipe(const std::string& pipeline_kind, if (model_version == "2" || model_version == "2.3") return PositiveOnlyRecipe(); } else if (pipeline_kind == "retake") { if (model_version == "2" || model_version == "2.5") return RetakeRecipe(model_version); + } else if (pipeline_kind == "a2vid_two_stage") { + // All four generations this table KEYS, mirroring the `t2a_one_stage` rows + // and for the same reason: `A2VidPipelineTwoStage` takes whatever + // `resolve_cli_params()` read off the checkpoint (a2vid_two_stage.py:311), + // exactly as `T2AOneStagePipeline` does at t2a_one_stage.py:178-179. There + // is no "which generations support audio-to-video" question upstream, so + // restricting these rows would be a local invention. This differs from + // `distilled_two_stage`'s TWO rows, which are two because two different + // references supply them. + // + // Four KEYS, not four params objects. `_PARAMS_SINCE_VERSION` + // (utils/constants.py:130-133) has rows for (2,4) and (2,3) only, falling + // through to LTX_2_PARAMS at :179 — so 2.5 has no params row of its own and + // resolves onto the 2.4 one, which is what `Ltx2DetectPipelineParams` above + // mirrors and says. The four keys exist because the RECIPE table refuses an + // unknown (kind, version) by name rather than defaulting. + if (model_version == "2") return A2VidTwoStageRecipe(Ltx2Params20(), kOmniNegativePrompt); + if (model_version == "2.3") return A2VidTwoStageRecipe(Ltx2Params23(), kOmniNegativePrompt); + if (model_version == "2.4") { + return A2VidTwoStageRecipe(Ltx2DetectPipelineParams("2.4"), LightricksNegativePrompt()); + } + if (model_version == "2.5") { + return A2VidTwoStageRecipe(Ltx2DetectPipelineParams("2.5"), LightricksNegativePrompt()); + } } else if (pipeline_kind == "t2a_one_stage") { if (model_version == "2") return T2aOneStageRecipe(Ltx2Params20(), kOmniNegativePrompt); if (model_version == "2.3") return T2aOneStageRecipe(Ltx2Params23(), kOmniNegativePrompt); diff --git a/src/vllm/multimodal/ltx2_video.cpp b/src/vllm/multimodal/ltx2_video.cpp index 7b4ff74bc..473de69e9 100644 --- a/src/vllm/multimodal/ltx2_video.cpp +++ b/src/vllm/multimodal/ltx2_video.cpp @@ -384,7 +384,7 @@ constexpr char kLtx2DurationHeadPathExtra[] = "duration_head_path"; // they are no longer trusted: the list below is derived from this file on every // run and compared, and the failure prints the replacement to paste in. // READER ANCHORS (derived and gated by test_ltx2_video): -// 809 819 820 882 978 994 996 1087 1112 1217 1258 1300 1302 +// 809 819 820 882 978 994 1029 1120 1145 1250 1291 1333 1335 const char* const kKnownLoadExtras[] = { kLtx2AudioPromptEmbedsExtra, kLtx2PipelineKindExtra, kLtx2ModelVersionExtra, @@ -993,6 +993,39 @@ std::unique_ptr Ltx2VideoEngine::Load(const VideoModelParams& p im.model_version = RecipeVersionKey(version); im.pipeline_kind = VideoExtra(params.extras, kLtx2PipelineKindExtra, "distilled_two_stage"); im.recipe = ResolveLtx2PipelineRecipe(im.pipeline_kind, im.model_version); + // ── the adapter a two-stage pipeline cannot run without (#1117) ─────────── + // + // `--distilled-lora` is `required=True` on the parser `A2VidPipelineTwoStage` + // selects (utils/args.py:1140-1153, reached through `default_2_stage_arg_parser` + // at `:1123` from a2vid_two_stage.py:311), and the reason is what stage 2 is: + // a THREE-sigma refinement (`:164`) that only the distilled weights can + // complete. Run it on a checkpoint carrying no adapter and it returns a clip + // of the right size, the right frame count and the right sample rate. + // + // Keyed on `recipe.requires_distilled_lora` rather than on the kind STRING, so + // the next recipe off this parser inherits it — `ti2vid_two_stages` (#1093) + // and `keyframe_interpolation` (#1096) are both already waiting. + // + // WHAT THIS CANNOT MIRROR, and it is filed rather than left here: + // `stage_2_loras = (*loras, *distilled_lora)` (`:114`) puts the adapter on + // stage 2 ALONE, against `loras=tuple(loras)` for stage 1 (`:107`). This + // engine fuses at load into ONE weight set, so the adapter reaches both + // phases. Owed by https://github.com/mudler/vllm.cpp/issues/1118. + if (im.recipe.requires_distilled_lora && + VideoExtra(params.extras, kLtx2LoraPathExtra).empty()) { + Fail("the '" + im.pipeline_kind + + "' pipeline needs a distilled LoRA and none was supplied. Upstream's " + "`--distilled-lora` is `required=True` on the parser this pipeline selects " + "(ltx-pipelines utils/args.py:1140-1153) and its stage 2 is a three-sigma " + "refinement (a2vid_two_stage.py:164) that the base weights were never distilled " + "for. Supply it through the '" + + std::string(kLtx2LoraPathExtra) + + "' load extra. Refused rather than rendered, because a distilled schedule on " + "undistilled weights returns a clip of the right size, frame count and sample rate. " + "NOTE the divergence this cannot express: upstream fuses that adapter into stage 2 " + "ALONE (a2vid_two_stage.py:114 against :107) and this engine fuses once at load, so " + "stage 1 sees it too — https://github.com/mudler/vllm.cpp/issues/1118."); + } im.max_phase = ExtraInt(params.extras, kLtx2MaxPhaseExtra, -1); if (im.max_phase >= static_cast(im.recipe.phases.size())) { Fail("the '" + std::string(kLtx2MaxPhaseExtra) + "' extra is " + @@ -1563,6 +1596,16 @@ void ApplyGuidanceOverrides(const std::map& extras, } return; } + // IGNORED, not refused, on a phase whose denoiser takes no params. This is + // `SimpleDenoiser` (utils/denoisers.py:3) and the a2vid stage 2 is the one + // phase in the table that reaches it: the flags exist on that pipeline's + // parser (a2vid_two_stage.py:311 -> utils/args.py:947-1006) and they reach + // stage 1's guider alone (`:233-236`), because stage 2 constructs + // `SimpleDenoiser(v_context_p, a_context_p)` (`:278`). Applying them here + // instead would switch on a guidance pass upstream's stage 2 does not run — + // and it would do it invisibly, since an extra forward changes no output + // shape, frame count or sample rate. + if (phase.denoiser == Ltx2PhaseDenoiser::kSimple) return; video->cfg_scale = ExtraDouble(extras, kLtx2VideoCfgScaleExtra, video->cfg_scale); video->stg_scale = ExtraDouble(extras, kLtx2VideoStgScaleExtra, video->stg_scale); video->rescale_scale = ExtraDouble(extras, kLtx2VideoRescaleScaleExtra, video->rescale_scale); @@ -2468,6 +2511,31 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { // already has through an encoder and a vocoder can only lose to it. Ltx2DecodedAudio a2v_source; const std::string a2v_audio_path = VideoExtra(gen.extras, kLtx2AudioPathExtra); + // REQUIRED on a recipe that says so (#1117). `--audio-path` is `required=True` + // (a2vid_two_stage.py:312-317), and the pipeline's whole shape is "denoise + // video AROUND this take": both stages freeze the audio stream on it + // (`:251-256`, `:291-296`) and the soundtrack handed back is the caller's own + // file (`:301-303`). + // + // Checked HERE and not at load, because `pipeline_kind` is a LOAD extra and + // `audio_path` is a per-generation one, so the question is only decidable once + // a request exists. Keyed on the recipe flag rather than on the kind string, + // for the reason `audio_only` gives in the header. + // + // WITHOUT THE TAKE THE RENDER STILL FINISHES. The audio stream is generated + // rather than supplied, which is ordinary joint generation, and the result is + // a clip of the right size with the right frame count and the right sample + // rate — indistinguishable from audio-to-video that ignored its input. + if (im.recipe.requires_audio_input && a2v_audio_path.empty()) { + Fail("the '" + im.pipeline_kind + "' pipeline is driven BY a waveform and no '" + + std::string(kLtx2AudioPathExtra) + + "' extra was supplied. Upstream's `--audio-path` is `required=True` " + "(ltx-pipelines a2vid_two_stage.py:312-317) and both of its stages freeze the audio " + "stream on the encoded take (`:251-256`, `:291-296`). Refused rather than rendered: " + "without it the soundtrack is GENERATED, and a generated one is a finished clip at the " + "right size, frame count and sample rate with nothing to show that the input was " + "ignored. Supply the take, or load with a `pipeline_kind` that generates audio."); + } if (!a2v_audio_path.empty()) { if (!im.has_audio_encoder) { Fail("'" + std::string(kLtx2AudioPathExtra) + "' names '" + a2v_audio_path + diff --git a/tests/vllm/models/test_ltx2_pipeline.cpp b/tests/vllm/models/test_ltx2_pipeline.cpp index 571d3f09f..cc9d079b6 100644 --- a/tests/vllm/models/test_ltx2_pipeline.cpp +++ b/tests/vllm/models/test_ltx2_pipeline.cpp @@ -3329,3 +3329,191 @@ TEST_CASE("ltx2 the res2s_two_stage recipe is upstream's HQ preset") { CHECK(Mentions(message, version)); } } + +// ─── LTX25-A2VID-RECIPE (#1117) ────────────────────────────────────────────── + +TEST_CASE("ltx2 a2vid: the recipe is upstream's TWO stages, not the distilled one") { + // EVERY FIELD HERE RENDERS WHETHER IT IS RIGHT OR WRONG. A wrong sigma set, a + // wrong downscale, a wrong stepper and a wrong guider all produce a finished + // clip of the right size, frame count and sample rate, so each is asserted + // against its own upstream anchor rather than against a neighbouring recipe. + const vllm::Ltx2PipelineRecipe a2v = + vllm::ResolveLtx2PipelineRecipe("a2vid_two_stage", "2.5"); + const vllm::Ltx2PipelineRecipe distilled = + vllm::ResolveLtx2PipelineRecipe("distilled_two_stage", "2.5"); + const vllm::Ltx2PipelineRecipe one = vllm::ResolveLtx2PipelineRecipe("one_stage", "2.5"); + REQUIRE(a2v.phases.size() == 2u); + + // ── stage 1 (a2vid_two_stage.py:225-258) ────────────────────────────────── + const vllm::Ltx2PhaseRecipe& s1 = a2v.phases[0]; + CHECK(s1.name == "stage_1"); + // `width // 2, height // 2` (:206-212), which is also what makes + // `assert_resolution(is_two_stage=True)` (:168) the 64-divisor arm. + CHECK(s1.spatial_downscale == 2); + CHECK(a2v.max_spatial_downscale() == 2); + CHECK(s1.input_transform == vllm::Ltx2PhaseInputTransform::kInitial); + // `self._scheduler.execute(steps=num_inference_steps)` (:225-227): DERIVED at + // run time. The distilled recipe's stage 1 carries a frozen 9-sigma list, and + // handing that to a full model that was never distilled is the difference + // between 30 steps and 8 on the 2.5 row resolved above — LTX_2_3_PARAMS sets + // num_inference_steps=30 (utils/constants.py:85) and 2.4, which 2.5 resolves + // onto, inherits it (:124). 40 is 2.0's own default (:47), not this row's. + CHECK(s1.sigmas.empty()); + CHECK(s1.use_official_sigma_schedule); + CHECK_FALSE(distilled.phases[0].sigmas.empty()); // the control for that claim + // `ModalitySpec.noise_scale` defaults to 1.0 (utils/types.py:110) and :247-250 + // sets none. #1013: at 0.0 the state stays as `create_initial_state` wrote it, + // which with no initial latent is all zeros, and a zero-initialised denoise + // still returns a finite clip. + CHECK(s1.noise_scale == 1.0); + // `:229-258` passes no `stepper`, so `EulerDiffusionStep()` applies + // (utils/blocks.py:526-527). The neighbouring distilled recipe selects the + // ANCESTRAL stepper on this very generation (distilled.py:76-84), which + // reaches a2vid through nothing, so the two are asserted side by side. + CHECK(s1.stepper == vllm::Ltx2StepperKind::kEuler); + CHECK(distilled.phases[0].stepper == vllm::Ltx2StepperKind::kEulerAncestral); + CHECK(s1.stepper_eta == 0.0); + CHECK(s1.noise_seed_offset == 0); + // `MultiModalGuider(params=video_guider_params, ...)` (:233-236), whose six + // fields are the params table's video row through the CLI defaults + // (utils/args.py:947-1006). Asserted against `one_stage`'s phase, which is + // built from the same row, so a change to the table moves both. + CHECK(s1.video_guidance.cfg_scale == one.phases[0].video_guidance.cfg_scale); + CHECK(s1.video_guidance.stg_scale == one.phases[0].video_guidance.stg_scale); + CHECK(s1.video_guidance.rescale_scale == one.phases[0].video_guidance.rescale_scale); + CHECK(s1.video_guidance.modality_scale == one.phases[0].video_guidance.modality_scale); + CHECK(s1.video_guidance.stg_blocks == one.phases[0].video_guidance.stg_blocks); + // ...and the values themselves, so this case still says which arm it is on if + // both recipes were changed together. `rescale_scale = 0.7` is what makes the + // x0-space question live (guiders.py:268-271). + CHECK(s1.video_guidance.cfg_scale == 3.0); + CHECK(s1.video_guidance.stg_scale == 1.0); + CHECK(s1.video_guidance.rescale_scale == 0.7); + CHECK(s1.video_guidance.modality_scale == 3.0); + + // THE AUDIO GUIDER IS THE DEFAULT ONE (:237-239) AND NOT THE TABLE'S ROW, and + // this is the field a reader is most likely to "fix" by symmetry with + // `OneStagePhase`, which takes the table's row and is right to + // (ti2vid_one_stage.py:215-218). A2Vid's audio stream is FROZEN, so the + // table's cfg 7.0 would buy an unconditional forward and a negative text + // encode for a delta multiplied into a latent the sampler cannot move. + CHECK(s1.audio_guidance.cfg_scale == 1.0); + CHECK(s1.audio_guidance.stg_scale == 0.0); + CHECK(s1.audio_guidance.rescale_scale == 0.0); + CHECK(s1.audio_guidance.modality_scale == 1.0); + CHECK(s1.audio_guidance.stg_blocks.empty()); + CHECK_FALSE(s1.audio_guidance.DoUnconditionalGeneration()); + CHECK_FALSE(s1.audio_guidance.DoPerturbedGeneration()); + CHECK_FALSE(s1.audio_guidance.DoIsolatedModalityGeneration()); + // The control: `one_stage` DOES take the table's audio row, so the assertions + // above are not passing because every recipe carries defaults. + CHECK(one.phases[0].audio_guidance.cfg_scale == 7.0); + // The CLI passes six video guider fields per request (:353-360). + CHECK(s1.allow_guidance_override); + + // ── stage 2 (a2vid_two_stage.py:277-297) ────────────────────────────────── + const vllm::Ltx2PhaseRecipe& s2 = a2v.phases[1]; + CHECK(s2.name == "stage_2"); + CHECK(s2.spatial_downscale == 1); + // `self.upsampler(video_state.latent[:1])` (:261). + CHECK(s2.input_transform == vllm::Ltx2PhaseInputTransform::kSpatialUpsample); + // `stage_2_sigmas: torch.Tensor = STAGE_2_DISTILLED_SIGMAS` (:164) — byte for + // byte the distilled recipe's stage 2, which is the ONE thing the two share. + CHECK(s2.sigmas == distilled.phases[1].sigmas); + CHECK_FALSE(s2.use_official_sigma_schedule); + // `noise_scale=stage_2_sigmas[0].item()` (:288). + REQUIRE_FALSE(s2.sigmas.empty()); + CHECK(s2.noise_scale == s2.sigmas.front()); + CHECK(s2.stepper == vllm::Ltx2StepperKind::kEuler); + // `SimpleDenoiser(v_context_p, a_context_p)` (:278) takes no params at all, so + // both guider fields stay at the positive-only defaults. + CHECK(s2.denoiser == vllm::Ltx2PhaseDenoiser::kSimple); + CHECK(s1.denoiser == vllm::Ltx2PhaseDenoiser::kGuided); + // AND the override is ALLOWED here, which is the pair a boolean alone cannot + // express. The guider flags DO exist on this pipeline's parser (`:311` selects + // `default_2_stage_arg_parser`), so a request carrying one is legal — it just + // reaches stage 1 and nothing else (`:233-236`). Refusing would reject a + // request upstream accepts; applying would switch on guidance upstream's + // stage 2 does not run, and `kSimple` above is what stops that. + CHECK(s2.allow_guidance_override); + // The control on the OTHER polarity: `distilled.py` selects + // `default_2_stage_distilled_arg_parser` (utils/args.py:1188), which never adds + // the flags at all, so both of its phases REFUSE — and they are `kSimple` too, + // which is exactly why the refusal has to be tested before the skip. + CHECK_FALSE(distilled.phases[1].allow_guidance_override); + CHECK(distilled.phases[1].denoiser == vllm::Ltx2PhaseDenoiser::kSimple); + CHECK_FALSE(s2.video_guidance.DoUnconditionalGeneration()); + CHECK_FALSE(s2.video_guidance.DoPerturbedGeneration()); + CHECK_FALSE(s2.video_guidance.DoIsolatedModalityGeneration()); + + // ── the recipe (a2vid_two_stage.py:143-166, utils/args.py:1123-1128) ─────── + CHECK(a2v.height == distilled.height); + CHECK(a2v.width == distilled.width); + CHECK(a2v.num_frames == distilled.num_frames); + CHECK(a2v.frame_rate == distilled.frame_rate); + CHECK(a2v.video_output_phase == 1); + CHECK(a2v.audio_output_phase == 1); + CHECK_FALSE(a2v.audio_only); + // Stage 1's schedule IS the step count (:226), so `--num-inference-steps` is + // honoured. The distilled recipe fixes both stages and refuses the override. + CHECK(a2v.allow_request_sigmas); + CHECK_FALSE(a2v.fixed_num_inference_steps); + CHECK(a2v.num_inference_steps == one.num_inference_steps); + CHECK_FALSE(distilled.allow_request_sigmas); // the control + // `:146` takes a negative prompt and `:183` reads `ctx_n` into the video + // guider's `negative_context`; the distilled recipe has no negative half at + // all, so both polarities are exercised here. + CHECK(a2v.allow_negative_prompt); + CHECK(a2v.negative_prompt == one.negative_prompt); + CHECK_FALSE(a2v.negative_prompt.empty()); + CHECK_FALSE(distilled.allow_negative_prompt); + CHECK_FALSE(a2v.allow_request_latents); + // `--audio-path` (:312-317) and `--distilled-lora` (utils/args.py:1140-1153) + // are BOTH `required=True`, and neither has a value this port can invent. + CHECK(a2v.requires_audio_input); + CHECK(a2v.requires_distilled_lora); + // The controls: no other recipe demands either, so a build that set the flags + // unconditionally is caught. + CHECK_FALSE(distilled.requires_audio_input); + CHECK_FALSE(distilled.requires_distilled_lora); + CHECK_FALSE(one.requires_audio_input); + CHECK_FALSE(one.requires_distilled_lora); +} + +TEST_CASE("ltx2 a2vid: all four generations resolve and nothing else does") { + // FOUR ROWS, mirroring the `t2a_one_stage` rows one for one and for the same + // reason: `A2VidPipelineTwoStage` takes whatever `resolve_cli_params()` read + // off the checkpoint (a2vid_two_stage.py:311), exactly as + // `T2AOneStagePipeline` does at t2a_one_stage.py:178-179. There is no "which + // generations support audio-to-video" question upstream, so restricting these + // rows would be a local invention. + for (const char* version : {"2", "2.3", "2.4", "2.5"}) { + INFO("version = ", std::string(version)); + CHECK_NOTHROW((void)vllm::ResolveLtx2PipelineRecipe("a2vid_two_stage", version)); + const vllm::Ltx2PipelineRecipe r = + vllm::ResolveLtx2PipelineRecipe("a2vid_two_stage", version); + REQUIRE(r.phases.size() == 2u); + CHECK(r.requires_audio_input); + CHECK(r.requires_distilled_lora); + CHECK(r.phases[0].spatial_downscale == 2); + CHECK(r.phases[0].stepper == vllm::Ltx2StepperKind::kEuler); + } + // The 2.4 and 2.5 rows take Lightricks' negative prompt and the older two take + // vLLM-Omni's, which is the split every other Lightricks-sourced row makes: + // the negative prompt travels with the GENERATION, not the pipeline. + CHECK(vllm::ResolveLtx2PipelineRecipe("a2vid_two_stage", "2.5").negative_prompt == + vllm::ResolveLtx2PipelineRecipe("one_stage", "2.5").negative_prompt); + CHECK(vllm::ResolveLtx2PipelineRecipe("a2vid_two_stage", "2").negative_prompt == + vllm::ResolveLtx2PipelineRecipe("one_stage", "2").negative_prompt); + CHECK(vllm::ResolveLtx2PipelineRecipe("a2vid_two_stage", "2.5").negative_prompt != + vllm::ResolveLtx2PipelineRecipe("a2vid_two_stage", "2").negative_prompt); + + // A version the table does not carry is REFUSED by name, never defaulted onto + // a neighbouring generation's guidance scales. + CHECK_THROWS((void)vllm::ResolveLtx2PipelineRecipe("a2vid_two_stage", "2.9")); + CHECK_THROWS((void)vllm::ResolveLtx2PipelineRecipe("a2vid_two_stage", "")); + // ...and so is the near-miss spelling, which is what a reader who knows the + // upstream FILE name rather than the pipeline kind would type. + CHECK_THROWS((void)vllm::ResolveLtx2PipelineRecipe("a2vid", "2.5")); + CHECK_THROWS((void)vllm::ResolveLtx2PipelineRecipe("a2v_two_stage", "2.5")); +} diff --git a/tests/vllm/multimodal/test_ltx2_video.cpp b/tests/vllm/multimodal/test_ltx2_video.cpp index 8758420eb..5d59d6bae 100644 --- a/tests/vllm/multimodal/test_ltx2_video.cpp +++ b/tests/vllm/multimodal/test_ltx2_video.cpp @@ -6984,3 +6984,361 @@ TEST_CASE("ltx2 guided video: the refusals that would otherwise RENDER (#1092)") CHECK(t.video_first_denoised == t.video_first_cond); } } + +// ─── LTX25-A2VID-RECIPE (#1117) ────────────────────────────────────────────── + +namespace { + +// An `a2vid_two_stage` engine on the shipped fixture. Both load-side +// requirements the recipe carries are met here: the spatial upsampler stage 2 +// needs (through `ConditioningParams`) and the distilled adapter upstream's +// `--distilled-lora required=True` demands. +vllm::multimodal::VideoModelParams A2VidParams(const ltx2_fixture::Paths& paths, + const std::string& lora) { + vllm::multimodal::VideoModelParams mp = ConditioningParams(paths); + mp.extras[vllm::multimodal::kLtx2PipelineKindExtra] = "a2vid_two_stage"; + mp.extras[vllm::multimodal::kLtx2LoraPathExtra] = lora; + return mp; +} + +// The request. Two things beside the take, and each is a property of the FIXTURE +// rather than of this row: +// +// * `steps = 2`, because stage 1's schedule is DERIVED from the step count +// (a2vid_two_stage.py:225-227) and two sigma intervals exercise the loop. +// That this is accepted at all is part of what the case asserts — the +// distilled recipe refuses a `steps` override. +// * the STG block list, because the reduced DiT has TWO blocks and the params +// row this recipe resolves names block 28 — LTX_2_3_PARAMS overrides 2.0's +// [29] to [28] (utils/constants.py:86) and 2.4, the row 2.5 resolves onto, +// inherits it (:124). `OneStageFixtureGuidance` carries the whole +// argument; the override reaching stage 1 and being IGNORED by stage 2 is +// itself gated below. +vllm::multimodal::VideoGenParams A2VidGen(const std::string& out_dir, const std::string& wav, + double start_time = 0.0) { + vllm::multimodal::VideoGenParams gen = FixtureGen(out_dir); + gen.steps = 2; + OneStageFixtureGuidance(&gen); + gen.extras[vllm::multimodal::kLtx2AudioPathExtra] = wav; + if (start_time != 0.0) { + gen.extras[vllm::multimodal::kLtx2AudioStartTimeExtra] = std::to_string(start_time); + } + return gen; +} + +// Every artifact a render wrote, concatenated. Downstream of the DiT weights and +// of every guidance decision, which is what makes it able to see a pass that ran +// on a phase the trace does not record. +std::string A2VidArtifacts(const std::string& out_dir, + const vllm::multimodal::VideoResult& result) { + std::string bytes; + for (int64_t f = 0; f < result.frame_count; ++f) { + char name[64]; + std::snprintf(name, sizeof(name), "/frame_%06lld.ppm", static_cast(f)); + bytes += ReadAll(out_dir + name); + } + bytes += ReadAll(std::string(result.audio_path)); + return bytes; +} + +} // namespace + +TEST_CASE("ltx2 a2vid: the pipeline renders through vllm.h and CONSUMES its take") { + // THE REACHABILITY CLAIM, and it is the point of this case rather than a note + // beside it. Entry point: `LoadVideoEngine` with a documented value of the + // documented `pipeline_kind` LOAD extra, then `Generate` with the documented + // `audio_path` per-generation extra. Nothing here constructs a recipe, a + // guider, a phase or a modality by hand. Deleting the `a2vid_two_stage` + // dispatch row in `ResolveLtx2PipelineRecipe` REDs this case at the load, + // which is what separates measuring a capability from measuring a class + // (.agents/reachability.md). + // + // `ltx2-gen --pipeline-kind a2vid_two_stage --audio-path ...` is the same two + // calls through the ABI, as a thin client that includes no internal header. + // The `/v1/videos` route CANNOT drive it: `VideoGenParamsFromRequest` never + // writes `gen.extras` (#928), so no per-generation extra reaches any engine + // over HTTP. Stated here because the reach claim has to exclude it. + Workspace ws; + const std::string lora = + WriteFixtureLora(ws.root + "/distilled.safetensors", kFixtureLoraTarget, 1.0F); + const std::string wav = WriteWav(ws.root + "/take.wav", 2, kFixtureAudioRate, 2.0); + + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(A2VidParams(ws.paths, lora)); + REQUIRE(engine != nullptr); + auto* ltx = dynamic_cast(engine.get()); + REQUIRE(ltx != nullptr); + CHECK(ltx->pipeline_kind() == "a2vid_two_stage"); + + const vllm::multimodal::VideoResult result = engine->Generate(A2VidGen(ws.root + "/a2v", wav)); + const vllm::multimodal::Ltx2ConditioningTrace t = ltx->last_conditioning(); + REQUIRE(t.completed); + CHECK(result.frame_count == 9); + // Upstream returns the caller's own waveform rather than a VAE round trip of + // it (`:301-303`), and the observable consequence is the SAMPLE RATE: the + // vocoder's BWE arm emits 48 kHz where the take went in at the audio VAE's own + // rate. + CHECK(result.sample_rate == kFixtureAudioRate); + + // ── the take was CONSUMED, not merely carried ───────────────────────────── + // + // A recipe-level assertion proves `noise_scale = 0.0` and `frozen = True` are + // SET. These four say the DiT saw the consequence, and they are read off the + // LAST phase — so stage 2's own `noise_scale` of 0.909375, which the loop + // applies to both streams, is inside what they measure. + CHECK(t.audio_conditioned); + CHECK_MESSAGE(t.audio_frozen, + "the audio denoise mask was not all zeros at the last phase, so the sampler was " + "free to move the caller's take (utils/types.py:104-106)"); + CHECK_MESSAGE(t.audio_sigma_max == 0.0, + "the scalar `Modality.sigma` was left at the schedule's value on some step; the " + "zeroed mask cannot reach that input, and a DiT told its clean conditioning is " + "noisy still renders"); + CHECK(t.audio_latent_absmax > 0.0); + CHECK(t.audio_latent_digest != 0); + + // THE CONTROL THAT MAKES THOSE MEAN SOMETHING. Same take, same request, a + // DIFFERENT seed: the audio latent must be BIT-IDENTICAL, because it is the + // encoded file and not a sample. A build that noised the audio stream — or + // that generated it and let the take decorate the trace — moves this digest, + // and moves nothing a caller can see. + vllm::multimodal::VideoGenParams reseeded = A2VidGen(ws.root + "/a2v_seed", wav); + reseeded.seed = 99; + (void)engine->Generate(reseeded); + const vllm::multimodal::Ltx2ConditioningTrace t_seed = ltx->last_conditioning(); + REQUIRE(t_seed.completed); + CHECK_MESSAGE(t_seed.audio_latent_digest == t.audio_latent_digest, + "the audio latent changed with the SEED, so it is being sampled rather than " + "taken from the caller's file"); + // ...and the second control, so the first cannot be passing because the latent + // is a constant: a different WINDOW of the same file gives a different latent. + const vllm::multimodal::VideoResult windowed = + engine->Generate(A2VidGen(ws.root + "/a2v_window", wav, 0.5)); + (void)windowed; + const vllm::multimodal::Ltx2ConditioningTrace t_window = ltx->last_conditioning(); + REQUIRE(t_window.completed); + CHECK_MESSAGE(t_window.audio_latent_digest != t.audio_latent_digest, + "windowing the take 0.5s later produced the SAME latent, so the samples are not " + "reaching the encoder"); + + // ── stage 1 ran upstream's GUIDED denoiser, in x0 space, on every arm ────── + // + // The trace's guided fields are recorded at step 0 of phase 0, which is + // a2vid's stage 1. Its guider is the params table's video row — cfg 3.0, + // stg 1.0, rescale 0.7, modality 3.0 — so all four passes run and the rescale + // branch, the one term that is NOT invariant between the two spaces, is live. + REQUIRE_MESSAGE(t.video_guided, "stage 1 did not go through the guided seam at all"); + CHECK(t.video_cond_forwards == 1); + CHECK(t.video_uncond_forwards == 1); + CHECK(t.video_perturbed_forwards == 1); + CHECK(t.video_modality_forwards == 1); + CHECK(t.video_guidance_cfg_scale == 3.0); + CHECK(t.video_guidance_stg_scale == 1.0); + CHECK(t.video_guidance_rescale_scale == 0.7); + CHECK(t.video_guidance_modality_scale == 3.0); + + // AND STAGE 1'S SCHEDULE WAS DERIVED, not read off a frozen list. + // `schedule_tokens` is written only on the branch that calls + // `Ltx2SigmaSchedule`, and stays 0 on a recipe carrying its own distilled + // sigmas — which is the difference between upstream's + // `self._scheduler.execute(steps=num_inference_steps)` (a2vid_two_stage.py:225-227) + // and the eight-step distilled list. Without this the recipe case is the only + // thing that can see a stage 1 handed the wrong schedule, and a wrong schedule + // renders. + CHECK_MESSAGE(t.schedule_tokens > 0, + "stage 1 did not derive its schedule from the step count, so it is running a " + "frozen sigma list upstream does not give it"); + + const size_t n = t.video_first_latent.size(); + REQUIRE(n > 0); + const size_t tokens = t.video_first_timesteps.size(); + REQUIRE(tokens > 0); + const size_t width = n / tokens; + REQUIRE(width * tokens == n); + // THE FIXTURE CAN DECIDE THIS AT ALL: `latent - sigma*velocity` and `velocity` + // coincide when the sample is zero. A REQUIRE, because nothing below + // discriminates once it fails. + double latent_span = 0.0; + for (const float x : t.video_first_latent) { + latent_span = std::max(latent_span, std::abs(static_cast(x))); + } + REQUIRE_MESSAGE(latent_span > 1e-3, "the step-0 sample is zero, so the two candidate tensors " + "coincide and nothing below discriminates"); + + struct Arm { + const char* name; + const std::vector& velocity; + const std::vector& x0; + }; + const Arm arms[] = { + {"cond", t.video_first_cond_velocity, t.video_first_cond}, + {"uncond", t.video_first_uncond_velocity, t.video_first_uncond}, + {"perturbed", t.video_first_perturbed_velocity, t.video_first_perturbed}, + {"modality", t.video_first_modality_velocity, t.video_first_modality}, + }; + for (const Arm& arm : arms) { + INFO("arm = " << std::string(arm.name)); + REQUIRE(arm.velocity.size() == n); + REQUIRE(arm.x0.size() == n); + // A zeroed velocity makes `to_denoised` the identity on this arm alone and + // would satisfy the equation while proving nothing. + double velocity_span = 0.0; + for (const float x : arm.velocity) { + velocity_span = std::max(velocity_span, std::abs(static_cast(x))); + } + REQUIRE_MESSAGE(velocity_span > 1e-6, "this arm's velocity is zero, so the equation below " + "holds for a reason that is not the one it tests"); + // `x0 = latent - sigma_token * velocity` (model.py:590-604), with the + // PER-TOKEN timestep and not the schedule scalar. + double residual = 0.0; + double against_velocity = 0.0; + for (size_t i = 0; i < n; ++i) { + const double sigma = static_cast(t.video_first_timesteps[i / width]); + const double expected = static_cast(t.video_first_latent[i]) - + sigma * static_cast(arm.velocity[i]); + residual = std::max(residual, std::abs(static_cast(arm.x0[i]) - expected)); + against_velocity = std::max( + against_velocity, + std::abs(static_cast(arm.x0[i]) - static_cast(arm.velocity[i]))); + } + INFO("max|x0 - (latent - sigma*v)| = " << residual); + INFO("max|x0 - velocity| = " << against_velocity); + // In VELOCITY space the first number is the whole sample and the second is + // exactly 0, which is what the RED prints. + CHECK(residual < 1e-4); + CHECK(against_velocity > 1e-6); + } +} + +TEST_CASE("ltx2 a2vid: every requirement the recipe adds refuses BY WHAT IS MISSING") { + // Three refusals, and each one guards a configuration that would otherwise + // RENDER — a finished clip at the right size, frame count and sample rate, + // with nothing in any output to show what was dropped. + Workspace ws; + const std::string lora = + WriteFixtureLora(ws.root + "/distilled.safetensors", kFixtureLoraTarget, 1.0F); + const std::string wav = WriteWav(ws.root + "/take.wav", 2, kFixtureAudioRate, 2.0); + + // ── no distilled adapter, refused at LOAD (utils/args.py:1140-1153) ──────── + { + vllm::multimodal::VideoModelParams mp = A2VidParams(ws.paths, lora); + mp.extras.erase(vllm::multimodal::kLtx2LoraPathExtra); + try { + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(mp); + FAIL_CHECK("an a2vid load with no distilled LoRA must be refused"); + } catch (const std::exception& e) { + const std::string message = e.what(); + INFO("message = " << message); + CHECK(message.find("distilled LoRA") != std::string::npos); + CHECK(message.find("lora_path") != std::string::npos); + CHECK(message.find("args.py:1140-1153") != std::string::npos); + // The divergence this refusal cannot repair is named in the same breath, + // so a reader who hits it is told where it is tracked. + CHECK(message.find("1118") != std::string::npos); + } + // THE CONTROL: the same load on the DEFAULT kind is fine without an adapter, + // so this is the recipe's requirement and not a new global one. + vllm::multimodal::VideoModelParams distilled = ConditioningParams(ws.paths); + CHECK_NOTHROW((void)vllm::multimodal::LoadVideoEngine(distilled)); + } + + // ── no take, refused at GENERATE (a2vid_two_stage.py:312-317) ────────────── + { + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(A2VidParams(ws.paths, lora)); + vllm::multimodal::VideoGenParams gen = A2VidGen(ws.root + "/no_take", wav); + gen.extras.erase(vllm::multimodal::kLtx2AudioPathExtra); + try { + (void)engine->Generate(gen); + FAIL_CHECK("an a2vid render with no audio_path must be refused"); + } catch (const std::exception& e) { + const std::string message = e.what(); + INFO("message = " << message); + CHECK(message.find("audio_path") != std::string::npos); + CHECK(message.find("a2vid_two_stage.py:312-317") != std::string::npos); + CHECK(message.find("GENERATED") != std::string::npos); + } + // THE CONTROL: the take is what the refusal is about, and supplying it on + // the same engine renders. + CHECK_NOTHROW((void)engine->Generate(A2VidGen(ws.root + "/with_take", wav))); + } + + // ── the guider override REACHES stage 1 and is IGNORED by stage 2 ────────── + // + // Upstream's `--video-cfg-guidance-scale` exists on this pipeline's parser + // (a2vid_two_stage.py:311 -> utils/args.py:947-1006) and reaches stage 1's + // guider alone (`:233-236`), because stage 2 is `SimpleDenoiser(...)` (`:278`). + // A build that REFUSED it would reject a request upstream accepts; a build + // that applied it to stage 2 would run an unconditional forward upstream's + // stage 2 does not, and neither shows up in any output. + { + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(A2VidParams(ws.paths, lora)); + auto* ltx = dynamic_cast(engine.get()); + REQUIRE(ltx != nullptr); + vllm::multimodal::VideoGenParams gen = A2VidGen(ws.root + "/override", wav); + gen.extras[vllm::multimodal::kLtx2VideoCfgScaleExtra] = "1.0"; + CHECK_NOTHROW((void)engine->Generate(gen)); + const vllm::multimodal::Ltx2ConditioningTrace t = ltx->last_conditioning(); + REQUIRE(t.completed); + // It reached STAGE 1: the trace's guidance fields are phase 0's, and the + // recipe's own value is 3.0. + CHECK_MESSAGE(t.video_guidance_cfg_scale == 1.0, + "the override did not reach stage 1's guider, so a2vid's caller-configured " + "guidance is unreachable"); + CHECK(t.video_uncond_forwards == 0); + + // AND IT DID NOT REACH STAGE 2, measured on the artifacts because no trace + // field records what the second phase did. + // + // The instrument is a pair of renders whose difference is a value that is + // ALREADY stage 1's. `video_stg_scale = 1.0` is exactly what this recipe's + // stage 1 carries (`utils/constants.py:52`), so applying it there changes + // nothing; stage 2's own STG scale is 0.0, so applying it THERE adds a + // perturbed forward per step and moves every pixel downstream of it. Equal + // bytes therefore mean the override stopped at stage 1, and that is a claim + // an `allow_guidance_override` boolean cannot make either way. + const vllm::Ltx2PipelineRecipe recipe = + vllm::ResolveLtx2PipelineRecipe("a2vid_two_stage", "2.5"); + REQUIRE(recipe.phases.size() == 2u); + REQUIRE_MESSAGE(recipe.phases[0].video_guidance.stg_scale == 1.0, + "the value below is no longer stage 1's own, so the two renders differ for " + "a second reason and the comparison proves nothing"); + REQUIRE(recipe.phases[1].video_guidance.stg_scale == 0.0); + + vllm::multimodal::VideoGenParams plain = A2VidGen(ws.root + "/stg_plain", wav); + const vllm::multimodal::VideoResult plain_result = engine->Generate(plain); + const std::string plain_bytes = A2VidArtifacts(ws.root + "/stg_plain", plain_result); + + vllm::multimodal::VideoGenParams restated = A2VidGen(ws.root + "/stg_restated", wav); + restated.extras[vllm::multimodal::kLtx2VideoStgScaleExtra] = "1.0"; + const vllm::multimodal::VideoResult restated_result = engine->Generate(restated); + const std::string restated_bytes = A2VidArtifacts(ws.root + "/stg_restated", restated_result); + + REQUIRE(plain_bytes.size() > 0); + REQUIRE(plain_bytes.size() == restated_bytes.size()); + // A COUNT of differing bytes, never the two buffers. These are PPM pixels + // and a WAV, so a failing `CHECK(a == b)` dumps raw binary into the report — + // which killed a mutation harness on the sibling row between applying a + // mutation and restoring it, and left the tree mutated. + size_t differing = 0; + for (size_t i = 0; i < plain_bytes.size(); ++i) { + if (plain_bytes[i] != restated_bytes[i]) ++differing; + } + CHECK_MESSAGE(differing == 0, + "restating stage 1's OWN stg_scale moved " << differing << " of " + << plain_bytes.size() + << " artifact bytes, so the override reached stage 2 — which runs " + "`SimpleDenoiser` upstream (a2vid_two_stage.py:278) and has no " + "guidance to switch on"); + // THE CONTROL for the same request on a recipe that FIXES its guidance: the + // distilled kind refuses the identical extra, so the acceptance above is + // this recipe's and not a weakening of that refusal. + const std::unique_ptr fixed = + vllm::multimodal::LoadVideoEngine(ConditioningParams(ws.paths)); + vllm::multimodal::VideoGenParams gen_fixed = FixtureGen(ws.root + "/override_fixed"); + gen_fixed.extras[vllm::multimodal::kLtx2VideoCfgScaleExtra] = "1.0"; + CHECK_THROWS((void)fixed->Generate(gen_fixed)); + } +}