fix(webrtc): reap checkpoint download workers

[gtong-nv]

Summary

• Prevent WebRTC shutdown from leaving checkpoint-download worker processes running after Ctrl+C.
• Terminate and reap multiprocessing children owned by each WebRTC server rank during shutdown.
• Escalate workers that do not exit within five seconds from terminate() to kill().
• Add a CPU regression test covering interruption while a checkpoint-download-style child process is active.

Problem

WebRTC startup preloads model checkpoints before the server begins accepting sessions. Sharded Hugging Face checkpoints are downloaded using multiprocessing workers. If the server is interrupted during this startup phase, the server rank can exit while those workers continue downloading as orphan processes re-parented to PID 1.

Implementation

The shared WebRTC bootstrap now performs child-process cleanup in its shutdown finally path for both the serving rank and distributed worker ranks. Cleanup:

1. Finds active multiprocessing children owned by the current rank.
2. Sends each child a termination request.
3. Waits up to five seconds for the children to exit and reaps them.
4. Kills and reaps any remaining children.

Rank 0 still sends the distributed exit signal even if child-process cleanup raises an error.

Testing

• pytest flashdreams/tests/test_webrtc_serving.py flashdreams/tests/test_webrtc_manager.py flashdreams/tests/test_webrtc_bootstrap.py -q — 10 passed
• Ruff check and format verification for the changed files — passed
• Targeted ty check for the changed files — passed

The repository-wide pre-commit type hook still reports the existing unresolved optional flip_evaluator import in flashdreams/flashdreams/quality/clip_compare.py; this is unrelated to this change.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[gtong-nv]

/ok to test 5c22865

[greptile-apps]

Greptile Summary

This PR adds child-process cleanup to the WebRTC shutdown path. The main changes are:

• Terminate and reap active multiprocessing children during shutdown.
• Escalate surviving child processes from terminate() to kill().
• Run cleanup from both serving and worker ranks.
• Add a regression test for interrupted startup with an active child process.

Confidence Score: 4/5

The shutdown cleanup needs one fix before merging.

• A stuck child can block rank 0 before it sends the distributed exit signal.
• Worker ranks can remain waiting even after the serving rank receives Ctrl+C.
• The new test covers the normal cleanup path, but not the stuck post-kill reap path.

flashdreams/flashdreams/serving/webrtc/bootstrap.py

Important Files Changed

Filename	Overview
flashdreams/flashdreams/serving/webrtc/bootstrap.py	Adds shutdown cleanup for multiprocessing children and restructures rank teardown; the final child reap can still block the distributed exit signal.
flashdreams/tests/test_webrtc_bootstrap.py	Adds a focused CPU regression test for interrupted WebRTC startup with an active child process.

_{Reviews (1): Last reviewed commit: "fix(webrtc): reap checkpoint download wo..." | Re-trigger Greptile}

[greptile-apps]

Exit Signal Waits Behind Reap

When a checkpoint download child survives terminate() and cannot be reaped after kill(), this unbounded join() can keep rank 0 inside cleanup forever. The later send_exit_signal() never runs, so worker ranks blocked in the distributed wait loop can remain alive after Ctrl+C.