Add shared realtime inference utilities for models to use
Summary
We need to simplify the process of models creating their own demos with flashdreams.
This should extract common realtime inference, runner I/O, presentation, timing, and acceleration utilities so multiple FlashDreams demos can share the same runtime building blocks instead of carrying duplicated local implementations.
The goal is not to force every model into one inference stack. Model-specific transformer math, schedulers, conditioning payloads, checkpoint remapping, renderers, and control schemas remain owned by their integrations. The shared code provides reusable primitives that demos can opt into.
Scope
We should provide shared utilities for:
- runner I/O: prompt resolution, image/video loading, normalization, output paths, MP4 writing, and stats JSON writing
- realtime serving primitives: normalized input/control state, frame publishing, media conversion, and presentation/backpressure helpers
- transport integration: WebRTC continues to own peer/session lifecycle while consuming transport-neutral realtime helpers
- MJPEG presentation: generic latest-frame and multipart stream plumbing, with demo-specific routes/UI kept local
- timing and profiling: shared milestone timestamp records, trace emission, rolling latency/perf summaries, and optional stage metadata
- acceleration policy: opt-in helpers for compile/CUDA-graph dispatch, prewarm, overlap, frame prefetch, and encoder lifecycle where behavior is generic
Design Principles
- Preserve the existing dependency direction:
core -> infra -> recipes/integrations.
- Keep
core model-agnostic and avoid integration-specific branches in shared layers.
- Keep model policy in integrations.
- Keep optional media/GPU dependencies lazy-imported.
- Preserve existing demo behavior during migration.
- Add parity checks for each extraction so code movement is behavior-preserving.
Non-Goals
Do not make CUDA graphs, native acceleration, FP8, a specific decoder, or a specific realtime protocol mandatory. Also do not rewrite all demos around a single concrete model API, although we can create an API later if needed.
Validation
Each migration is validated with focused CPU/GPU tests, import/compile checks, and demo-specific parity checks where applicable.
Existing demos should be manually validated to confirm they are still working and have not been impacted either in quality or performance.
Add shared realtime inference utilities for models to use
Summary
We need to simplify the process of models creating their own demos with flashdreams.
This should extract common realtime inference, runner I/O, presentation, timing, and acceleration utilities so multiple FlashDreams demos can share the same runtime building blocks instead of carrying duplicated local implementations.
The goal is not to force every model into one inference stack. Model-specific transformer math, schedulers, conditioning payloads, checkpoint remapping, renderers, and control schemas remain owned by their integrations. The shared code provides reusable primitives that demos can opt into.
Scope
We should provide shared utilities for:
Design Principles
core -> infra -> recipes/integrations.coremodel-agnostic and avoid integration-specific branches in shared layers.Non-Goals
Do not make CUDA graphs, native acceleration, FP8, a specific decoder, or a specific realtime protocol mandatory. Also do not rewrite all demos around a single concrete model API, although we can create an API later if needed.
Validation
Each migration is validated with focused CPU/GPU tests, import/compile checks, and demo-specific parity checks where applicable.
Existing demos should be manually validated to confirm they are still working and have not been impacted either in quality or performance.