Stabilize Omnidreams WebRTC Manifest Demo#393
Open
gtong-nv wants to merge 7 commits into
Open
Conversation
Signed-off-by: Gangzheng Tong <gtong@nvidia.com>
- VideoEncoder Protocol + ChunkDeliveryResult in
flashdreams/serving/webrtc/encoders.py, with two implementations:
DefaultRTCEncoder (adapter over aiortc's software encoder) and
PyNvHardwareEncoder (NVENC H.264 via PyNvVideoCodec, output as
av.Packet).
- NVENCVideoTrack (media.py) delivers pre-encoded packets on aiortc's
public av.Packet -> H264Encoder.pack() path -- no reach into
aiortc private attributes. Drop-oldest overflow with a bounded
Queue[av.Packet]; recv() paces at 1/fps like BufferedVideoTrack.
- select_encoder() factory with a two-stage capability probe:
Stage 1 GetEncoderCaps (silent fallback to DefaultRTCEncoder on
environmental "not supported"; loud EncoderInitError under
backend='nvenc'); Stage 2 CreateEncoder (hard error, never a
silent fallback -- masking a driver / session-pool / hardware
failure would hide real problems).
- NVENC configured for interactive low-latency H.264: fmt=ABGR
(NV_ENC_BUFFER_FORMAT_ABGR is word-ordered -> little-endian byte
layout [R,G,B,A]; see _chunk_to_abgr_cuda_frames docstring),
preset=P4, ULL tuning, CBR rate control, bf=0, lookahead=0,
repeatspspps=1 (aiortc's H264Encoder.pack does not synthesize SPS
and PPS). Packets carry pts on the RTP 90 kHz clock so
H264Encoder.pack rescales cleanly via convert_timebase.
- PyNvVideoCodec>=2.1,<3 added as a hard dep on
integrations/omnidreams -- omnidreams already requires CUDA at
runtime, so the install matrix is unchanged.
- Startup emits a single INFO log line per session,
'Video encoder ready: backend={pynvvideocodec|aiortc} ...',
giving one grep anchor regardless of which backend was selected.
- ci_cpu tests cover: Protocol conformance, factory branch coverage
(Stage-1 library / caps / bounds failures under both auto and nvenc,
Stage-2 hard-error regression guard under both), pre-encoded packet
plumbing on NVENCVideoTrack (pts / time_base / drop-oldest overflow),
cross-chunk packet-ordering invariant that guards the manager's
sequential-await pattern from a future create_task rewrite, and
compat guards for aiortc + PyNvVideoCodec public surfaces.
- ci_gpu smoke test encodes a 4-frame chunk on real hardware and
asserts Annex-B start code + IDR + SPS + PPS in the first packet,
plus pts=0 and time_base=1/90000.
No runtime path currently consumes the new abstraction; behavior
change lands in the follow-up commit.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Contributor
Greptile SummaryThis PR adds Omnidreams manifest support to the WebRTC serving path. The main changes are:
Confidence Score: 5/5This looks safe to merge.
Important Files Changed
Reviews (3): Last reviewed commit: "Stabilize Omnidreams WebRTC presentation" | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
… fallback
- OmnidreamsRuntimeConfig gains encoder_backend
(auto | nvenc | default), encoder_bitrate_bps, encoder_gop.
select_encoder() runs on the runtime executor thread inside
_initialize_video_encoder_sync; the encoder is closed cleanly in
_close_sync so its NVENC session slot is released promptly.
- _generate_one_chunk_sync drops the .cpu() D2H copy on the tensor
and replaces it with torch.cuda.current_stream().synchronize().
Same wall-clock cost as before (both wait for the compute stream to
drain), but the hardware encoder can now read the CUDA tensor
directly via DLPack; the software path picks up the D2H inside
BufferedVideoTrack's worker.
- omnidreams.webrtc.server exposes one new CLI flag,
--prefer_sw_encoder. Defaults false -> encoder_backend='auto'
(probe NVENC and fall back silently to aiortc's software encoder
on Stage-1 unsupported). Setting it maps to encoder_backend='default'
and skips the NVENC probe entirely -- useful for A/B profiling and
known-flaky NVENC hosts. The tri-state config field stays on
OmnidreamsRuntimeConfig for programmatic use and test coverage of
the 'nvenc' loud-on-failure branch.
- Manager wiring: addTransceiver + setCodecPreferences constrain the
SDP to H.264 when the encoder emits pre-encoded packets
(prefers_codec == 'h264'). Post-answer, transceiver._codecs is
inspected; if H.264 did not land (e.g. a browser that will not
offer it), _enforce_h264_or_fallback closes the NVENC session and
installs DefaultRTCEncoder + BufferedVideoTrack via
sender.replaceTrack -- a pre-stream swap, no renegotiation.
- Chunk delivery in the generation worker goes through
encoder.deliver_chunk(...). The await is load-bearing for
cross-chunk packet ordering (guarded by the regression test added
in the previous commit): a create_task rewrite here would allow
chunks to complete out of order and packets to land on the track
with non-monotonic pts.
- Test suite extends _FakeVideoEncoder to satisfy the VideoEncoder
Protocol, threads video_encoder through every ManagedWebRTCSession
construction, and adds ci_cpu tests for _enforce_h264_or_fallback
(swap on VP8-negotiated, keep on H.264-negotiated, swap on empty
codec list) plus a parametrized test that --prefer_sw_encoder
correctly maps to encoder_backend ('default' when set, 'auto' when
unset).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- encoders.py: guard the ``PyNvVideoCodec`` import under ``TYPE_CHECKING`` so ty sees a single ``Any``-typed ``nvc`` name. Without the guard, ty preserved a ``<module PyNvVideoCodec> | None | Any`` union and rejected the ``None`` fallback and every attribute access on ``GetEncoderCaps`` / ``FORCEIDR`` / ``CreateEncoder``. Runtime path (the ``else:`` branch) keeps the same guarded try / except behaviour. - encoders.py: narrow the ``VideoEncoder.create_track`` return type from ``MediaStreamTrack`` to ``BufferedVideoTrack | NVENCVideoTrack`` so the manager can assign the result to the ``ManagedWebRTCSession .video_track`` field without a widening error. - manager.py: change ``ManagedWebRTCSession.video_track`` annotation from ``MediaStreamTrack`` to ``BufferedVideoTrack | NVENCVideoTrack``. The aiortc base type does not advertise ``.close`` / ``.fps`` / ``.qsize``, which the generation worker and liveness watchdog need. Drop the now-unused ``MediaStreamTrack`` import. - test_encoders.py: swap mypy-style ``# type: ignore[misc|arg-type]`` to ty-style ``# ty:ignore[invalid-assignment|invalid-argument-type]`` on the frozen-dataclass and SimpleNamespace test sites. Add ``assert packet.pts is not None`` before ``int(packet.pts)`` in the ordering tests -- ``av.Packet.pts`` is nullable at the type level even though the fake encoder always sets it. - test_webrtc_manager.py: add a local ``_FakeVideoEncoder`` stub and thread ``video_encoder=`` through the ``ManagedWebRTCSession`` construction the shared base test uses. - integrations/lingbot/tests/test_webrtc_runtime.py: same treatment for the three ``_ManagedLingbotSession`` constructions -- the lingbot tests broke when we promoted ``video_encoder`` to a required field on the shared ``ManagedWebRTCSession`` base. - integrations/omnidreams/tests/test_webrtc_runtime.py: drop 8 unused ``# ty:ignore[invalid-argument-type]`` comments ty flagged (5 on ``video_encoder=…``, 3 on ``transceiver=transceiver``). - integrations/omnidreams/ludus-renderer/examples/render_mirror_augmented.py: drop the ``# ty:ignore[unresolved-import]`` on the ``PyNvVideoCodec`` import. The comment was pre-existing but became unused when this feature promoted PyNvVideoCodec from an optional extra to a hard dependency of ``integrations/omnidreams``. - Ruff format touched a few files in passing (test_nvenc_track.py, test_nvenc_smoke.py, test_encoders.py); those changes are formatting-only and included here. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: Gangzheng Tong <gtong@nvidia.com>
Signed-off-by: Gangzheng Tong <gtong@nvidia.com>
Signed-off-by: Gangzheng Tong <gtong@nvidia.com>
gtong-nv
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stabilize Omnidreams WebRTC Manifest Demo
Before:
Browser can only present at ~20 FPS even the server can produces 30 FPS steady.
After:
Both Browser and server can maintain 30+FPS.
Summary
Adds manifest-driven Omnidreams WebRTC session setup and stabilizes the interactive browser playback path. The branch also brings in the NVENC H.264 WebRTC encoder path, richer runtime/browser profiling, and fixes manifest override tracking for abbreviated long CLI options.
The WebRTC demo can now prefer hardware H.264 encode when the browser negotiates it, fall back or fail loudly according to the selected encoder mode, and expose enough server/client timing to distinguish model, encode, transport, decode, and compositor bottlenecks.
Changes
cli_args.py.VideoEncoderabstraction withDefaultRTCEncoderand PyNvVideoCodec-backed NVENC H.264 support.auto,nvenc, ordefaultencoder backends, with--prefer_sw_encoderand--require_nvencCLI behavior.Notes
Media FPSfromPresentedFPS.Media FPSreflects receive/decode throughput;Presentedreflects browser compositor cadence.