Extract realtime input and media primitives#385
Merged
Conversation
Add a transport-neutral serving.realtime package for keyboard/input state, latest-frame publishing, tensor/array RGB conversion, and lazy JPEG encoding. Refactor WebRTC controls and media to consume the shared primitives while preserving existing import compatibility and BufferedVideoTrack behavior. Add CPU tests covering input snapshots, frame-bus shutdown semantics, media conversion parity, and JPEG encoding.
Contributor
Greptile SummaryThis PR extracts a transport-neutral
Confidence Score: 5/5Safe to merge; all changed paths behave correctly under the documented contracts and the bfloat16/uint8-validation regressions flagged in earlier rounds are resolved. The refactoring is mechanical and well-tested. The one behavioral change worth noting — scene-change in MJPEGStreamingPresenter no longer immediately wakes blocked stream handlers — adds up to one second of transition latency but does not affect correctness or data integrity. No changed path introduces wrong data, dropped state, or broken cleanup. streaming_presenter.py — the scene-change wakeup removal is the only non-trivial behavioral delta in the PR. Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart TD
subgraph realtime["serving.realtime (new)"]
FB["frame_bus.py\nLatestFrameBus"]
IN["input.py\nKeyboardState / KeyboardResampler\nCameraPoseIntegrator\nSparseInputSnapshot"]
ME["media.py\n_as_numpy / _scale_rgb\nrgb_frame_to_uint8\ntensor_chunk_to_rgb_frames\nencode_rgb_frame_to_jpeg"]
end
subgraph webrtc["serving.webrtc (refactored)"]
WC["controls.py\n(re-export shim)"]
WM["media.py\n(re-export shim)"]
MGR["manager.py"]
BVT["BufferedVideoTrack"]
end
subgraph flashvsr["flashvsr integrations"]
SV["grpc/streaming_view.py"]
UC["grpc/uplift_client.py"]
end
subgraph omnidreams["omnidreams integrations"]
KB["interactive_drive/input/keyboard.py"]
SP["interactive_drive/streaming_presenter.py\nMJPEGStreamingPresenter"]
GU["grpc/utils.py"]
end
IN --> WC
ME --> WM
ME --> SV
ME --> UC
ME --> GU
ME --> SP
IN --> KB
FB --> SP
WC --> MGR
WM --> BVT
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
subgraph realtime["serving.realtime (new)"]
FB["frame_bus.py\nLatestFrameBus"]
IN["input.py\nKeyboardState / KeyboardResampler\nCameraPoseIntegrator\nSparseInputSnapshot"]
ME["media.py\n_as_numpy / _scale_rgb\nrgb_frame_to_uint8\ntensor_chunk_to_rgb_frames\nencode_rgb_frame_to_jpeg"]
end
subgraph webrtc["serving.webrtc (refactored)"]
WC["controls.py\n(re-export shim)"]
WM["media.py\n(re-export shim)"]
MGR["manager.py"]
BVT["BufferedVideoTrack"]
end
subgraph flashvsr["flashvsr integrations"]
SV["grpc/streaming_view.py"]
UC["grpc/uplift_client.py"]
end
subgraph omnidreams["omnidreams integrations"]
KB["interactive_drive/input/keyboard.py"]
SP["interactive_drive/streaming_presenter.py\nMJPEGStreamingPresenter"]
GU["grpc/utils.py"]
end
IN --> WC
ME --> WM
ME --> SV
ME --> UC
ME --> GU
ME --> SP
IN --> KB
FB --> SP
WC --> MGR
WM --> BVT
Reviews (5): Last reviewed commit: "Fix Phase 2 lint issues" | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Validate RGB channel shape for bvtchw media chunks and reject inconsistent uint8 value_range arguments instead of silently returning raw bytes. Tighten realtime serving tests by removing the tautological WebRTC media self-comparison, adding validation coverage for the new media errors, and making the LatestFrameBus close-wakeup test deterministically block before close() is called.
Promote floating torch tensors to float32 before numpy conversion in the shared realtime media helper, preserving the old WebRTC converter behavior for bfloat16 model outputs. Add regression coverage for bfloat16 TCHW and BVTCHW tensor chunks.
Route OmniDreams interactive keyboard, MJPEG frame handoff, and JPEG encoding through the shared realtime input, frame bus, and media helpers. Reuse the shared JPEG encoder in FlashVSR and OmniDreams gRPC paths, and add focused CPU coverage for the migrated behavior.
Apply the ruff import-order fixes required by the CPU pre-commit job and keep the new Phase 2 files on the full Apache license header style.
gtong-nv
reviewed
Jul 20, 2026
| sync_device=sync_device, | ||
| ) | ||
| output = io.BytesIO() | ||
| image_from_array(rgb_uint8).save(output, format="JPEG", quality=quality) |
Collaborator
There was a problem hiding this comment.
maybe we can add the nvjpeg path here if that HW unit is available ?
Collaborator
Author
There was a problem hiding this comment.
Good idea, I will add this to my TODO list.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add a transport-neutral serving.realtime package for keyboard/input state, latest-frame publishing, tensor/array RGB conversion, and lazy JPEG encoding. Refactor WebRTC controls and media to consume the shared primitives while preserving existing import compatibility and BufferedVideoTrack behavior.
Add CPU tests covering input snapshots, frame-bus shutdown semantics, media conversion parity, and JPEG encoding.