Skip to content

Add hardware-accelerated WebRTC video encoding with CPU fallback #379

Description

@gtong-nv

Add hardware-accelerated WebRTC video encoding with CPU fallback

Summary

The Omnidreams and LingBot WebRTC presentation path currently appears to use CPU/software encoding after copying generated frames back to host memory. We should prefer a hardware/GPU-accelerated encode path when available, falling back in order from NVENC to a CUDA implementation, and using the current CPU path only as the final fallback.

Current behavior

  • Both WebRTC runtimes detach and copy generated chunks to CPU before enqueueing them for presentation:
    • integrations/omnidreams/omnidreams/webrtc/session.py: video_chunk.detach().cpu()
    • integrations/lingbot/lingbot/webrtc/session.py: video_chunk.detach().cpu()
  • The shared WebRTC media path converts tensors to NumPy RGB frames and creates PyAV frames with VideoFrame.from_ndarray(frame_array, format="rgb24") in flashdreams/flashdreams/serving/webrtc/media.py.
  • The WebRTC manager adds the track directly with peer_connection.addTrack(video_track) and does not appear to select hardware codec preferences or an NVENC/CUDA-backed encoder.

This means generation can be GPU-bound, while presentation still pays GPU-to-host copy plus CPU color conversion/encoding costs.

Desired behavior

  • Detect available hardware/GPU encoding support at startup or session creation.
  • Prefer NVENC where available and compatible with browser/session negotiation, especially H.264 and/or AV1 on supported NVIDIA GPUs.
  • If NVENC is unavailable or cannot be negotiated, fall back to a CUDA-based implementation before using CPU/software encoding.
  • Use the existing CPU/software path only when no suitable NVENC or CUDA path is available, negotiation fails, or the platform does not support GPU acceleration.
  • Expose clear logging/telemetry for the selected presentation encoder path, for example:
    • webrtc_encoder=nvenc_h264
    • webrtc_encoder=nvenc_av1
    • webrtc_encoder=cuda
    • webrtc_encoder=software

Notes / open questions

  • We may need to evaluate whether aiortc/PyAV can be configured cleanly for hardware encoding in this path, or whether a separate GPU encode pipeline is needed before feeding WebRTC.
  • The intended fallback order should be: NVENC first, CUDA implementation second, CPU/software last.
  • Omnidreams already converts generated frames to uint8 before the CPU copy; LingBot currently relies on the shared media converter to scale [-1, 1] tensors to uint8 on CPU. A hardware path should avoid unnecessary host conversion where possible.
  • CPU/software encoding should remain available as the reliable fallback path.

Acceptance criteria

  • WebRTC presentation can use NVENC when available and negotiated successfully.
  • When NVENC is unavailable, WebRTC presentation can fall back to a CUDA implementation before falling back to CPU/software.
  • The current CPU/software path remains functional as the final fallback.
  • The selected encoder path is visible in logs or session telemetry.
  • Omnidreams and LingBot both use the shared behavior consistently.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Fields

    No fields configured for Enhancement.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions