[lori-ren]

@coderabbitai summary

Description

An audit of who creates, reads and releases multimodal items (request params, encoder-cache entries, encoder and chunk views, prefetch outputs and events) on the inline, item-scheduled and side-stream prefetch paths. Two gaps were confirmed by tests that fail on main and are fixed here; the other transitions already hold and are now pinned by tests.

• Prefetch stream order. Side-stream prefetch stages a request's MRoPE positions and embeddings on its own CUDA stream. The forward stream sliced the MRoPE positions during input preparation before any wait, and the prefetch event was dropped when the first prefill chunk had no multimodal rows, so a later chunk read the embedding unordered. Input preparation now waits for the event at the point where it is handed over, and calls record_stream on the request's CUDA tensors so that post-prefill cleanup cannot hand their storage back to the side stream while forward-stream reads are still queued.
• Items behind a reused KV prefix. The EAGER policy binds a waiting request's items from prompt position zero. When KV reuse then starts the prefill past such an item, the item kept its encoder-cache reference, and with a tight output budget the request could never schedule its first chunk. Item selection now releases the items behind the prefill cursor first.

The PR also drops the stale note that inline encoder caching supports only single-modality requests (the two serve schema snapshots are regenerated for it), and documents the prefetch stream rule and the reference-release points in the multimodal developer guide.

Found and left unchanged:

• The main and side streams share the vision tower's attention workspace when a prefetch encode overlaps an inline encode. A targeted overlap test on H100 did not produce a mismatch, so no cross-stream fence is added.
• With KV cache manager V1 and MAX_UTILIZATION, a recompute pause releases multimodal inputs the replay needs. V1 is deprecated, and [#17882][fix] Preserve multimodal data for MAX_UTILIZATION pause replay #18113 addresses it.

Test Coverage

• tests/unittest/_torch/pyexecutor/engine/runners/decoder/test_inputs.py: prefetch-staged MRoPE positions and embeddings are read after the prefetch event, and staged storage outlives the request while it is still read. These fail without the inputs.py change.
• tests/unittest/_torch/pyexecutor/test_multimodal_scheduler.py: an EAGER item behind a reused prefix does not block the first chunk (fails without the scheduler.py change); termination between chunks releases each remaining reference once.
• tests/unittest/_torch/multimodal/test_mm_encoder_cross_iter_prefetch.py and tests/unittest/_torch/pyexecutor/test_recompute_pause.py: cancel after prefetch, and a V2 recompute pause keeping multimodal inputs.
• MMMU on Qwen3-VL-2B with chunked prefill and KV cache manager V2, for the item, EAGER, inline and prefetch paths: no difference from the unmodified baseline beyond run-to-run noise.

The MMMU runs were done on H100 before the branch was rebased onto current main. The unit tests were rerun after the rebase, against binaries built before it. The serve schema snapshots have not been regenerated since the rebase.

PR Checklist

Please review the following before submitting your PR:

• PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

• PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

• Test cases are provided for new code paths (see test instructions)

• If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

• Any new dependencies have been scanned for license and vulnerabilities

• CODEOWNERS updated if ownership changes

• Documentation updated as needed

• Update tava architecture diagram if there is a significant design change in PR.

• The reviewers assigned automatically/manually are appropriate for the PR.

• Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.