DNI: render profile repro

integrations/omnidreams/README.md

@@ -80,6 +80,9 @@ From the workspace root, run:
 uv run --package flashdreams-omnidreams torchrun --nproc_per_node 1 -m omnidreams.webrtc.server --pipeline_config_name omnidreams-sv-2steps-chunk2-loc6-lightvae-lighttae-perf --scene-uuid 065dcac9-ee67-4434-a835-c6b816c88e48 --port 8089
 ```
 
+For issue 195 renderer/serving profiling, see
+[docs/issue195_renderer_profile_repro.md](docs/issue195_renderer_profile_repro.md).
+
 When `--scene_dir` is omitted, the server downloads the selected scene from the
 configured Hugging Face org, extracts its `clipgt-<uuid>.usdz` archive, and
 stages it under `FLASHDREAMS_CACHE_DIR` (or `~/.cache/flashdreams`). If

integrations/omnidreams/docs/issue195_renderer_profile_repro.md

@@ -0,0 +1,105 @@
+# Issue 195 Renderer Profiling Repro
+
+This branch includes temporary profiling instrumentation for OmniDreams WebRTC
+serving. It records whether end-to-end serving time is spent in the model
+pipeline, queue/enqueue logic, or Ludus render conditioning.
+
+## Base
+
+The original repro was run from:
+
+```text
+upstream/main
+3816e32d99a001eab91996f1e52fe488be1ee9cf
+```
+
+Use this branch directly after it is published. No patch application is needed.
+
+## Launch On HSG
+
+Set `HF_TOKEN` or `HF_TOKEN_FILE` first. Then request a full GB200 node, but run
+the WebRTC server with one visible GPU, matching the OmniDreams WebRTC README
+path.
+
+```bash
+cd /lustre/fsw/portfolios/nvr/projects/nvr_torontoai_videogen/users/junchenl/flashdreams
+git fetch <remote> issue-195-render-profile-repro
+git switch issue-195-render-profile-repro
+
+uv sync --package flashdreams-omnidreams --extra interactive-drive
+
+srun -A nvr_torontoai_videogen -p batch --qos=interactive \
+  -N1 --gpus-per-node=4 --ntasks=1 --cpus-per-task=16 --time=02:00:00 \
+  --chdir "$PWD" \
+  bash -lc '
+    export OMNIDREAMS_LUDUS_RENDER_PROFILE=1
+    export OMNIDREAMS_LUDUS_RENDER_PROFILE_CUDA_EVENTS=1
+    unset OMNIDREAMS_LUDUS_MSAA_SAMPLES
+    CUDA_VISIBLE_DEVICES=0 uv run --no-sync --package flashdreams-omnidreams \
+      torchrun --nproc_per_node 1 -m omnidreams.webrtc.server \
+      --pipeline_config_name omnidreams-sv-2steps-chunk2-loc6-lightvae-lighttae-perf \
+      --scene-uuid 065dcac9-ee67-4434-a835-c6b816c88e48 \
+      --port 8095 2>&1 | tee issue195_webrtc_profile.log
+  '
+```
+
+`OMNIDREAMS_LUDUS_MSAA_SAMPLES` is intentionally unset so the renderer uses its
+default `4` samples. The profile log should include `renderer_msaa_samples: 4.0`.
+
+## Forward And Drive The Viewer
+
+In another shell, replace `<allocated-node>` with the Slurm node name:
+
+```bash
+socat -d -d TCP-LISTEN:8097,reuseaddr,fork,bind=0.0.0.0 \
+  TCP:<allocated-node>.cm.cluster:8095
+```
+
+Open:
+
+```text
+http://oci-hsg-cs-001-vscode-01:8097/request_session
+```
+
+Click Connect Session and drive for 20-30 seconds. The first few chunks may
+include warmup or compile noise, so summarize later chunks.
+
+## Summarize
+
+```bash
+python integrations/omnidreams/scripts/summarize_webrtc_profile.py \
+  issue195_webrtc_profile.log \
+  --min-chunk 4
+```
+
+Use `--session latest` for the most recent browser run, or `--session all` to
+include loopback warmup and browser sessions together.
+
+## Expected Signature
+
+The model pipeline is fast enough for more than 40 FPS, but the end-to-end
+viewer generation loop is around 18-20 FPS because render conditioning dominates.
+
+Observed browser-session summary from the original repro:
+
+```text
+selected_chunks=31 session=latest min_chunk=4
+gen_ms                             avg=  439.3 ms / 8 frames
+wrapper_render_condition_ms        avg=  259.8 ms
+renderer_ctx_render_ms             avg=  258.6 ms
+ctx_render_plugin_cuda_ms_sum      avg=  254.9 ms
+pipeline_total_ms                  avg=  175.0 ms
+pipeline_total_ms_wo_finalize      avg=  126.2 ms
+enqueue_ms                         avg=   26.4 ms
+```
+
+Interpretation:
+
+```text
+wrapper_render_condition_ms ~= renderer_ctx_render_ms
+renderer_ctx_render_ms      ~= ctx_render_plugin_cuda_ms_sum
+```
+
+So the extra serving time is inside `self.ctx.render(...)`, and nearly all of
+that time is accumulated by Ludus CUDA render plugin calls across the 8
+conditioning frames.

integrations/omnidreams/ludus-renderer/ludus_renderer/_ops/context.py

@@ -15,6 +15,7 @@
 
 """Ludus rendering context - ``LudusCudaTimestampedContext``."""
 
+import time
 from typing import List, Optional, Tuple
 
 import torch
@@ -27,6 +28,24 @@
 )
 
 
+def _record_ms_summary(
+    profile: dict[str, float], prefix: str, values: list[float]
+) -> None:
+    if not values:
+        return
+    profile[f"{prefix}_count"] = float(len(values))
+    profile[f"{prefix}_sum"] = float(sum(values))
+    profile[f"{prefix}_avg"] = float(sum(values) / len(values))
+    profile[f"{prefix}_min"] = float(min(values))
+    profile[f"{prefix}_max"] = float(max(values))
+
+
+def _create_event_profiler():
+    from flashdreams.infra.profiler import EventProfiler
+
+    return EventProfiler()
+
+
 def _compute_element_aabbs(
     vertices: torch.Tensor, prefix_sum: torch.Tensor, device: torch.device
 ) -> torch.Tensor:
@@ -77,6 +96,9 @@ def __init__(self, device=None):
         self._cameras: List[FThetaCamera] = []
         self._camera_intrinsics: Optional[torch.Tensor] = None
         self.needs_vflip = False  # CUDA renders top-down (standard image convention)
+        self.enable_render_profiling = False
+        self.enable_render_profile_cuda_events = False
+        self.last_render_profile: dict[str, float] = {}
 
         # Per-scene flat buffers
         self._scenes: List[dict] = []
@@ -424,21 +446,47 @@ def render(
         """
         assert self._camera_intrinsics is not None, "call upload_cameras first"
 
+        total_t0 = time.perf_counter()
         n = scene_ids.shape[0]
         all_images = []
+        profile_enabled = bool(getattr(self, "enable_render_profiling", False))
+        cuda_event_enabled = (
+            profile_enabled
+            and bool(getattr(self, "enable_render_profile_cuda_events", False))
+            and scene_ids.is_cuda
+            and torch.cuda.is_available()
+        )
+        profile: dict[str, float] = {}
+        scalar_item_ms: list[float] = []
+        query_prep_ms: list[float] = []
+        plugin_host_ms: list[float] = []
+        event_profiler = _create_event_profiler() if cuda_event_enabled else None
+
+        if profile_enabled:
+            profile["ctx_render_batch_size"] = float(n)
 
         plugin = _get_plugin()
 
+        loop_t0 = time.perf_counter()
         for i in range(n):
+            scalar_item_t0 = time.perf_counter()
             sid = int(scene_ids[i].item())
             cid = int(camera_ids[i].item())
             ts_val = int(timestamps_us[i].item())
             cam_type = int(camera_type_ids[i].item())
+            if profile_enabled:
+                scalar_item_ms.append((time.perf_counter() - scalar_item_t0) * 1e3)
 
+            query_prep_t0 = time.perf_counter()
             sd = self._scenes[sid]
             cam_intrinsics = self._camera_intrinsics[cid : cid + 1].contiguous()
             pose = camera_poses[i : i + 1].contiguous()
+            if profile_enabled:
+                query_prep_ms.append((time.perf_counter() - query_prep_t0) * 1e3)
 
+            if event_profiler is not None:
+                event_profiler.record(f"before_plugin_{i}")
+            plugin_t0 = time.perf_counter()
             img = plugin.ludus_render_fwd_cuda_timestamped(
                 self.cpp_wrapper,
                 sd["timestamps"],
@@ -459,6 +507,46 @@ def render(
                 resolution,
                 self._tessellation_threshold,
             )
+            if event_profiler is not None:
+                event_profiler.record(f"plugin_{i}")
+            if profile_enabled:
+                plugin_host_ms.append((time.perf_counter() - plugin_t0) * 1e3)
             all_images.append(img)
 
-        return torch.cat(all_images, dim=0)
+        if profile_enabled:
+            profile["ctx_render_loop_host_ms"] = (time.perf_counter() - loop_t0) * 1e3
+            _record_ms_summary(
+                profile, "ctx_render_scalar_item_host_ms", scalar_item_ms
+            )
+            _record_ms_summary(profile, "ctx_render_query_prep_host_ms", query_prep_ms)
+            _record_ms_summary(profile, "ctx_render_plugin_host_ms", plugin_host_ms)
+
+        if event_profiler is not None:
+            event_profiler.record("before_cat")
+        cat_t0 = time.perf_counter()
+        output = torch.cat(all_images, dim=0)
+        if event_profiler is not None:
+            event_profiler.record("cat")
+        if profile_enabled:
+            profile["ctx_render_cat_host_ms"] = (time.perf_counter() - cat_t0) * 1e3
+
+        if event_profiler is not None:
+            sync_t0 = time.perf_counter()
+            cuda_elapsed_ms = event_profiler.sync_and_summarize()
+            profile["ctx_render_cuda_event_sync_host_ms"] = (
+                time.perf_counter() - sync_t0
+            ) * 1e3
+            _record_ms_summary(
+                profile,
+                "ctx_render_plugin_cuda_ms",
+                [cuda_elapsed_ms[f"plugin_{i}"] for i in range(n)],
+            )
+            profile["ctx_render_cat_cuda_ms"] = cuda_elapsed_ms["cat"]
+
+        if profile_enabled:
+            profile["ctx_render_total_host_ms"] = (time.perf_counter() - total_t0) * 1e3
+            self.last_render_profile = profile
+        else:
+            self.last_render_profile = {}
+
+        return output