Extract shared realtime acceleration helpers#389
Open
jarcherNV wants to merge 8 commits into
Open
Conversation
Add infra.acceleration.frame_prefetch with CUDA host prefetch, lazy CUDA-frame materialization, and generic prefetch dispatch helpers. Migrate the OmniDreams raster and world-model lazy frame paths to share the implementation while keeping the old cuda_host_prefetch import path as a compatibility shim.
Contributor
Greptile SummaryThis PR extracts reusable acceleration helpers (frame prefetch, CUDA graph dispatch, prewarm, overlap runners, encoder lifecycle) from demo-specific code into a shared
Confidence Score: 5/5Safe to merge; all previously flagged issues (prefetch exception recovery, copy=False protocol, control-flow inversion, gRPC finalization race) are addressed and the shared helpers are well-tested. The extraction is clean: each new module centralises one responsibility, existing callers are migrated faithfully, and the CPU test suite covers the new helpers. Remaining notes are minor redundancies and an error-message regression, none of which affect runtime behaviour. integrations/omnidreams/omnidreams/grpc/server.py — confirm the full-render serialisation matches the intended multi-session concurrency contract. flashdreams/flashdreams/recipes/template/transformer/init.py — diagnostic error message is less specific after the migration. Important Files Changed
Reviews (8): Last reviewed commit: "Address realtime acceleration review fee..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Add an acceleration helper for CUDA-graph capture thresholds and conditional/unconditional CUDAGraphWrapper dispatch. Migrate Wan, Cosmos, Template, OmniDreams, and LingBot reset/dispatch paths to use the helper while preserving existing compatibility attributes. Also clear helper-owned graph wrappers when OmniDreams native FP8 releases the PyTorch network so the previous memory-release behavior is preserved.
Collaborator
Author
|
/ok to test 338fb9e |
Collaborator
Author
|
/ok to test 338fb9e |
Add shared acceleration helpers for timed model prewarm steps, CUDA graph prewarm sequence counts, prewarm deadlines, and warmup-index exclusion. Migrate OmniDreams interactive-drive model/session warmup timing and FlashVSR distributed compile prewarm loops onto the shared helpers while keeping WebRTC transport warmup separate.
Preserve the pipeline_factory warmup path when offload_text_encoder is enabled. Only defer model construction for the non-factory offload path, matching the original warmup semantics.
Add shared acceleration helpers for synchronous, host-thread, and CUDA-stream overlap work. Migrate OmniDreams gRPC KV-cache finalization to the host-thread runner while preserving the existing wait-before-next-request semantics.
Narrow the fake CUDA event before accessing test-only methods so the overlap tests pass ty in the CPU CI lint job.
Add shared acceleration helpers for config-backed one-shot encoder setup, tensor-to-CPU precompute results, and encoder release with CUDA cache cleanup. Migrate Wan and OmniDreams encoder release paths plus the OmniDreams interactive-drive offload path onto the shared helper.
jarcherNV
changed the title
Recover from failed lazy-frame prefetch materialization, expose the shared frame-prefetch helpers from the acceleration package, and serialize the gRPC render/finalization scheduling path to avoid concurrent overlap submissions.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Extract shared realtime acceleration helpers
Summary
This PR extracts reusable realtime acceleration helpers from demo-specific code
and migrates the existing demos/integrations to use the shared implementations.
It builds on the shared realtime serving, timing, and presentation work by
adding common infrastructure for frame prefetching, CUDA graph dispatch policy,
prewarm execution, overlapped work, and one-shot encoder lifecycle management.
Changes
materialization, and generic prefetch dispatch.
cuda_host_prefetchimport path as a compatibility shim.execution.
materialization, release, and CUDA memory cleanup.
acceleration utilities.
Validation
interactive-drive flows.
pre-refactor phase 5 baseline.