Problem
The vLLM backend in the official Triton image
(nvcr.io/nvidia/tritonserver:*-vllm-python-py3) only supports the image
modality. In backends/vllm/utils/request.py, the only multimodal branch is
multi_modal_data={"image": ...} — there is no video or audio support.
So even though the vLLM engine supports them, Triton users can't pass audio/video.
Request
Please add vLLM-Omni support to the
official Triton vLLM backend, so Triton can natively serve omni-modal models
(e.g. Qwen3-Omni) with video and audio in multi_modal_data, not just images.
An official omni example model_repository would also help.
Current workaround
Custom Python pre-processing that extracts video frames with ffmpeg and sends
them as multiple images — loses native temporal/audio handling and inflates tokens.
Context
Problem
The vLLM backend in the official Triton image
(
nvcr.io/nvidia/tritonserver:*-vllm-python-py3) only supports theimagemodality. In
backends/vllm/utils/request.py, the only multimodal branch ismulti_modal_data={"image": ...}— there is novideooraudiosupport.So even though the vLLM engine supports them, Triton users can't pass audio/video.
Request
Please add vLLM-Omni support to the
official Triton vLLM backend, so Triton can natively serve omni-modal models
(e.g. Qwen3-Omni) with
videoandaudioinmulti_modal_data, not just images.An official omni example
model_repositorywould also help.Current workaround
Custom Python pre-processing that extracts video frames with ffmpeg and sends
them as multiple images — loses native temporal/audio handling and inflates tokens.
Context
tritonserver:26.06-vllm-python-py3, Qwen3-VL-8B, single-node GPU