Add hardware-accelerated WebRTC video encoding with CPU fallback
Summary
The Omnidreams and LingBot WebRTC presentation path currently appears to use CPU/software encoding after copying generated frames back to host memory. We should prefer a hardware/GPU-accelerated encode path when available, falling back in order from NVENC to a CUDA implementation, and using the current CPU path only as the final fallback.
Current behavior
- Both WebRTC runtimes detach and copy generated chunks to CPU before enqueueing them for presentation:
integrations/omnidreams/omnidreams/webrtc/session.py: video_chunk.detach().cpu()
integrations/lingbot/lingbot/webrtc/session.py: video_chunk.detach().cpu()
- The shared WebRTC media path converts tensors to NumPy RGB frames and creates PyAV frames with
VideoFrame.from_ndarray(frame_array, format="rgb24") in flashdreams/flashdreams/serving/webrtc/media.py.
- The WebRTC manager adds the track directly with
peer_connection.addTrack(video_track) and does not appear to select hardware codec preferences or an NVENC/CUDA-backed encoder.
This means generation can be GPU-bound, while presentation still pays GPU-to-host copy plus CPU color conversion/encoding costs.
Desired behavior
- Detect available hardware/GPU encoding support at startup or session creation.
- Prefer NVENC where available and compatible with browser/session negotiation, especially H.264 and/or AV1 on supported NVIDIA GPUs.
- If NVENC is unavailable or cannot be negotiated, fall back to a CUDA-based implementation before using CPU/software encoding.
- Use the existing CPU/software path only when no suitable NVENC or CUDA path is available, negotiation fails, or the platform does not support GPU acceleration.
- Expose clear logging/telemetry for the selected presentation encoder path, for example:
webrtc_encoder=nvenc_h264
webrtc_encoder=nvenc_av1
webrtc_encoder=cuda
webrtc_encoder=software
Notes / open questions
- We may need to evaluate whether aiortc/PyAV can be configured cleanly for hardware encoding in this path, or whether a separate GPU encode pipeline is needed before feeding WebRTC.
- The intended fallback order should be: NVENC first, CUDA implementation second, CPU/software last.
- Omnidreams already converts generated frames to
uint8 before the CPU copy; LingBot currently relies on the shared media converter to scale [-1, 1] tensors to uint8 on CPU. A hardware path should avoid unnecessary host conversion where possible.
- CPU/software encoding should remain available as the reliable fallback path.
Acceptance criteria
- WebRTC presentation can use NVENC when available and negotiated successfully.
- When NVENC is unavailable, WebRTC presentation can fall back to a CUDA implementation before falling back to CPU/software.
- The current CPU/software path remains functional as the final fallback.
- The selected encoder path is visible in logs or session telemetry.
- Omnidreams and LingBot both use the shared behavior consistently.
Add hardware-accelerated WebRTC video encoding with CPU fallback
Summary
The Omnidreams and LingBot WebRTC presentation path currently appears to use CPU/software encoding after copying generated frames back to host memory. We should prefer a hardware/GPU-accelerated encode path when available, falling back in order from NVENC to a CUDA implementation, and using the current CPU path only as the final fallback.
Current behavior
integrations/omnidreams/omnidreams/webrtc/session.py:video_chunk.detach().cpu()integrations/lingbot/lingbot/webrtc/session.py:video_chunk.detach().cpu()VideoFrame.from_ndarray(frame_array, format="rgb24")inflashdreams/flashdreams/serving/webrtc/media.py.peer_connection.addTrack(video_track)and does not appear to select hardware codec preferences or an NVENC/CUDA-backed encoder.This means generation can be GPU-bound, while presentation still pays GPU-to-host copy plus CPU color conversion/encoding costs.
Desired behavior
webrtc_encoder=nvenc_h264webrtc_encoder=nvenc_av1webrtc_encoder=cudawebrtc_encoder=softwareNotes / open questions
uint8before the CPU copy; LingBot currently relies on the shared media converter to scale[-1, 1]tensors touint8on CPU. A hardware path should avoid unnecessary host conversion where possible.Acceptance criteria