LTX-2.5 full-model DiT forward measured at ~162 s on GB10; 60 forwards will not fit a 4 h lease

[localai-bot]

First measured DiT forward cost for LTX-2.5 on the full 21.004 B model: ~162 s per forward on GB10. At the structural 60 forwards that is ~2.7 h of denoise alone, which does not fit a 4 h lease once load and tail are included.

This is also the first run in which a DiT forward is known to have executed at all. Nine prior attempts never reached one.

Provenance

Run 20260819T150230Z, job d985d423, dgx:gpu0 (GB10, 20 cores, 119.631 GiB).
binary_sha256=0a43a750deb3457520fb10ed44ee0cb62db8740dca65cc4e2d7c33864ae456b2, built_from=7b9e207b1 (carries #1252).
Config: one_stage --device cuda, 1024x576/25f, full/dev transformer, STG off and modality 1.0 (the device arm refuses a perturbed forward, #1092), real CFG live.

The measurement

[t=2417s] governor[gpu-edge(no-stacks)]: first_dit=481s observed_forward_starts=3
          per_forward~69.1s projected_total=5225s budget=7153s
[t=3251s] ABORT[92] PROJECTED OVERRUN: first_dit=481s per_forward~162.0s x 60
          + tail 600s = 10803s > budget 7153s
[t=3256s] memory: baseline_used=4.752GiB peak_used=79.658GiB OWN_DEMAND=74.907GiB
          min_avail=39.97GiB
          child exit=-15 samples=3192 gpu_busy_idle_edges=7

• First forward at t=481 s. Against the pre-fix(LTX25-TEXT-LINEAR-SEAM): the caption projection is one core for 671 s per pass, so route it through vt::MatmulBT #1252 conditioning stretch of 1740 s / 1738 s / 3002 s, so #1252 is confirmed end-to-end on real weights: time-to-first-forward fell from ~29-50 min to ~8 min.
• Per-forward ~162 s, and 60 forwards is structural (ltx2_pipeline.cpp:521/524/527 — 30 steps × 2 for CFG, not adjustable from the CLI).
• Memory is a non-issue: OWN_DEMAND=74.907 GiB on a box that started at 4.752 GiB used, min_avail=39.97 GiB against a 12 GiB floor.

Note the estimate's own trajectory: per_forward read 69.1 s at three observed starts and 162.0 s by the abort. An early projection from few samples understated it by 2.3×. Anyone re-measuring should let the observation count grow before quoting a figure.

What is NOT established

• Attribution. phase=OTHER throughout — eu-stack unwinds zero frames in the rc worker container, so no stack-based attribution was possible. 162 s is a wall-clock interval between GPU busy→idle edges, not a profile. Whether the time is GPU compute, host-side glue, or transfer is unknown.
• dit_runs=0 is the stack-based counter and is not evidence of absence; gpu_busy_idle_edges=7 is the counter that worked.
• No comparison against vLLM/vLLM-Omni on the same workload. Without that, 162 s is a fact about this port, not a parity gap.
• Nothing about smaller geometries; the fallback rung had not reported when this was filed.

Why it matters

one_stage is currently the only full-model arm that reaches generation, and at 162 s/forward it needs ~3 h of denoise for a 25-frame 1024x576 clip. Every full-model render is therefore blocked on a bounded lease unless the per-forward cost comes down or the geometry shrinks.

The obvious next step is attribution: get a working profile inside the lease (an in-process phase marker, since eu-stack is blind there — #1010's territory), then compare against the pinned oracle on an identical workload.

Related: #1252 and #1259 (the two GEMM-seam fixes that made reaching this point possible), #1254, #1092 (the perturbation refusal that forces STG off on the device arm).

[localai-bot]

Correction: the structural forward count is 120, not 60 — and this issue's projection doubles with it

Measured while implementing #1092 (PR #1428).

This issue records "60 forwards is structural (ltx2_pipeline.cpp:521/524/527 — 30 steps × 2 for CFG, not adjustable from the CLI)". That was true only of the configuration the render was forced into, not of the model.

The 60 was 30 × 2 because the device arm's perturbation refusal pinned the other two guiders to identity — STG and the isolated-modality pass were switched off, so their legs never ran. At LTX-2.5's own defaults (stg_scale 1.0, modality_scale 3.0, ltx2_pipeline.cpp:951-959) the count is 30 × 4 = 120.

What that does to the projection

	forwards	denoise at ~162 s/forward
as measured here (guidance off)	60	~2.7 h
at the model's defaults (#1428)	120	~5.4 h

The 1024x576 rung already aborted on a projected overrun at 60 forwards (10,803 s against a 7,153 s budget). At 120 it is roughly twice that, so this issue is now the binding constraint on whether a correctly-guided full-model render is reachable at all in a bounded lease — not merely on how large it can be.

Upstream shares no work between the legs

Checked rather than assumed: _guided_denoise batches the legs into one call at batch n·B, and BatchSplitAdapter splits that back into sequential chunks — which is exactly what this port already runs. So there is no batching win available by mirroring upstream more closely; the four legs really are four forwards.

Consequence for anyone reading this issue

Any plan sized on 60 forwards is stale. A correctly-guided render at 1024x576/25f projects to roughly 5.4 h of denoise alone before load, conditioning, VAE decode and tail. Either the per-forward cost comes down, or correctly-guided full-model renders stay confined to small geometries.

Attribution of the 162 s remains open and is still the first thing to do — it is a wall-clock interval between GPU busy→idle edges, not a profile. eu-stack unwinds zero frames in the rc worker container, so the in-process phase markers landing via #1408/#1419 (last= per forward) are what will finally break it down.

Also worth noting for whoever takes this: device peak memory does not increase with the perturbed legs — a perturbed self-attention allocates no qb/qn/kb/kn/attn, and a skipped cross direction allocates none of its six. Read from the allocations in #1428, not measured on hardware.

[localai-bot]

Measured directly: 16.43 s per forward, not 162 s. This issue's central number was an artefact of the instrument it warned about.

First run with #1413's in-process per-forward timing (last=), on the full 21.004 B model.

Run: 20260820T084103Z, job bac9f4c5, dgx:gpu0 (GB10). binary_sha256=bb8e0961eef9a439c458eb9bc80e5f94cff716a5f7982d242bcc5d37b2fbdd3a, built_from=b537a5344 — verified to contain #1092 (7f9c6802e), #1010 (89261c955) and #1413 (d0eff4f25), with the old perturbation refusal absent from the tree (0 occurrences of cannot run one / Refusing rather than running an unperturbed).

Configuration: one_stage --device cuda, 768x448/25f, and — for the first time — guidance at the model's own defaults: all four zeroing flags dropped, stg_scale 1.0, modality_scale 3.0.

The measurement

dit forward 1  phase 0 step 1/30  t=478.846s
dit forward 2  phase 0 step 1/30  t=495.695s last=16.848s
dit forward 3  phase 0 step 1/30  t=512.112s last=16.418s
dit forward 4  phase 0 step 1/30  t=528.443s last=16.331s
dit forward 5  phase 0 step 2/30  t=544.873s last=16.429s
dit forward 6  phase 0 step 2/30  t=561.444s last=16.571s
dit forward 7  phase 0 step 2/30  t=578.058s last=16.614s
dit forward 8  phase 0 step 2/30  t=594.328s last=16.269s

n=7, min 16.269 s, median 16.429 s, max 16.848 s — a 3.5% spread, which is what a real per-forward cost looks like.

Why the old figure was wrong

162.0 s came from the governor's gpu-edge(no-stacks) estimator: a wall-clock interval between GPU busy→idle transitions, observed from outside the process because eu-stack unwinds zero frames in the rc worker container. That counter reads 0 when the GPU is saturated — there is no idle transition to count — so it was never sampling forwards. Its own trajectory showed the instability: it read 69.1 s at 3 observed starts and 162.0 s by the abort, a 2.3x drift on the same run.

Adjusting for geometry does not rescue it. 768x448 is 0.583x the pixels of 1024x576, so the scaled expectation from 162 s would be ~94 s. The measured 16.43 s is ~5.7x lower than that.

Consequences, and they are large

1. The four guidance legs are confirmed structurally. Forwards 1-4 are step 1/30, forwards 5-8 are step 2/30 — four legs per step, 120 forwards, exactly as #1092 predicted.

2. Guided full-model rendering is comfortably feasible. At 16.43 s x 120 forwards, denoise is ~33 min, on top of first_dit=478.8 s. That is roughly 41 minutes plus VAE decode and tail, inside a 4 h lease with large margin — where this issue previously projected 10,803 s against a 7,153 s budget and aborted the rung.

3. The 1024x576 abort was based on a bad number. Scaling the measured cost by pixels gives ~28 s/forward there, so 120 forwards is ~56 min — very likely feasible, and worth re-attempting rather than treated as out of reach.

4. The "5.4 h ceiling" for guided rendering is void. It was 60 x 162 s doubled. Both inputs are superseded: the count is 120 (exact) and the cost is 16.43 s (measured).

What this does not establish

• The wall multiplier for guidance is still not isolated, because this run has no unguided 768x448 comparison taken with the same instrument. The unguided run measured 2990 s total wall at 60 forwards; a like-for-like per-forward comparison needs the same last= lane on an unguided run.
• Attribution is still open. 16.43 s is a measured interval around the forward, not a profile. What inside it is GPU compute versus host glue versus transfer remains unknown.
• No oracle comparison on an identical workload, so this is a fact about this port, not a parity gap.

The governor's total_forwards and its projection arithmetic should be re-derived from last= rather than from GPU edges — the edge estimator should be treated as unreliable wherever the GPU is saturated, which is the normal case.

[localai-bot]

The guided DiT forward has a measured cost law, and it bounds what a lease can render

Three rungs of one guided render (92bfedf4, run 20260820T223701Z, full
checkpoint, dgx:gpu0/GB10, 30 steps x 4 legs = 120 structural forwards) give a
per-forward cost that fits a power law in the token count
tokens = (W/32) * (H/32) * ceil(F/8):

geometry	tokens	measured s/forward	fitted	err
768x448:25f	1344	16.61 (n=119)	16.53	-0.5%
1024x576:49f	4032	135.77 (n=3)	137.62	+1.4%
1024x576:97f	7488	458.24 (n=3)	454.29	-0.9%

s/forward = 1.52e-05 * tokens^1.929

The middle row was predicted at 138.6 s before that rung ran and measured
135.77 s, so the law is validated out of sample rather than fitted to its own
points. Two independent fits (mine and the render agent's) agree on the exponent
to four digits and on the coefficient to 0.3%.

What it means

The exponent is ~1.93, which is the DiT attention term. Cost is therefore set by
the total token count and nothing else - pixels and frames are perfectly
interchangeable, and "make it bigger" and "make it longer" spend from the same
quadratic budget.

Consequence for a 4 h lease: after ~1,100 s build and ~465 s first_dit, the
per-forward budget is ~95 s, which is a ceiling of ~3,300 tokens.

geometry	tokens	denoise	reachable on a clean 4 h lease?
768x448:49	2352	1.6 h	yes
896x512:49	3136	2.8 h	yes - the largest rung that fits
1024x576:49	4032	4.6 h	no
1024x576:97	7488	15.1 h	no
1024x576:121	9216	22.6 h	no

So a full lease buys roughly 2 seconds of video at ~0.45 MP. The 4-second
1024x576 clip that has been the informal target is a 23 h render, not a long
one - it was never a lease-sized job, and no amount of scheduling makes it one.

Why this belongs on this row

This row owns the attribution of the per-forward interval. The law says how the
interval scales; it says nothing about what the interval is made of. The
16.61 s at 1344 tokens is still an interval, not a profile - the split between
GPU compute, host glue and transfer is unmeasured, and that split is the only
lever that changes the table above. A 2x reduction in the coefficient moves the
4 h ceiling from ~3,300 to ~4,800 tokens, which is 1024x576:49.

Recorded so the next render opens directly at its largest reachable rung instead
of measuring its way down a ladder. In this run the descent spent ~2,800 s on
rungs the closed form rejects in zero seconds, which is about 40% of a landing
rung.