Skip to content

perf(editor): stream short Studio Sound cuts with bounded history - #3254

Merged
richiemcilroy merged 1 commit into
CapSoftware:mainfrom
vedanth-jadhav:perf/stream-studio-sound-short-cuts
Oct 25, 2026
Merged

richiemcilroy merged 1 commit into
CapSoftware:mainfrom
vedanth-jadhav:perf/stream-studio-sound-short-cuts

Conversation

@vedanth-jadhav

@vedanth-jadhav vedanth-jadhav commented Oct 25, 2026 •

Copy link
Copy Markdown

Short forward cuts with Studio Sound currently fall back to decoding every recording track into full PCM before MP4 export can start. This avoids the preroll read-back error fixed by #2812, but makes a short edit of a long recording wait for a full-track allocation/decode.

Keep a bounded history for enhanced microphone reads so the voice enhancer can warm up after those cuts while its decoder continues forward. Ordinary tracks request zero history. The original sample arithmetic, gain/stereo handling, enhancer reset rules, cancellation and full-file validation remain in use. Backward source timelines, retiming, music and unsupported transition shapes retain their existing eligibility rules.

This implements the bounded-history extension suggested in review. #3229 adds end-to-end regression coverage for short cuts; this extends the streaming implementation itself.

Complete MP4 measurements

Synthetic one-hour recording edited to eight seconds with a 100 ms forward gap; two four-second clips. The benchmark follows the GPUI app's ExporterBuilder::build_for_mp4 → Mp4ExportSettings::export_prepared path, renders and encodes the complete video, mixes Studio Sound and validates the remaining source audio before successful completion.

Source Profile Preparation, s First frame progress, s Total export, s Peak RSS, MiB
WAV s 0.776 → 0.096 0.890 → 0.207 2.309 → 1.760 1539.2 → 224.2
WAV editor3 0.750 → 0.084 0.856 → 0.179 2.161 → 1.647 1539.1 → 223.6
AAC s 2.049 → 0.078 2.156 → 0.184 3.587 → 3.013 1539.6 → 233.9
AAC editor3 1.962 → 0.077 2.050 → 0.184 3.355 → 2.937 1539.3 → 234.2

Complete exports take 12.4–23.8% less time, first rendered-frame progress arrives 76.7–91.5% sooner, and process peak RSS is 84.8–85.5% lower. The AAC case removes about 1.9 seconds of initial waiting. These are standalone exporter-process measurements; they do not include an already-open editor's retained audio or establish total desktop-app memory savings.

Five alternating independent processes per revision/source/profile/case, with warm filesystem and shader caches. All 120 MP4 exports have matching decoded video and float-PCM SHA-256 hashes within each workload, across baseline/head and profiles. Timers include actual builder preparation, rendering, audio processing/validation, encoding and final output completion. First progress is the callback after a rendered frame is received, before forwarding it to the encoder; it is not UI window presentation. Metric columns are independent process medians.

The export_prepared phase alone is 9–100% slower in the newly eligible cases because full-file validation moves into that phase; the reported total includes preparation on both revisions. Validation is still performed on the entire source. The change does not promise constant total export time independent of source length.

Already-streaming exact splits and one-second gaps are controls. Their total times range from 2.5% faster to 1.3% slower; worst slowdown is WAV/size, 1.724 → 1.746 s. Peak RSS control changes are at most about 1.7 MiB. The largest preparation-control slowdown is 5.8 ms. No control speedup is claimed.

Linux x86_64, AMD EPYC 9V74, two-CPU quota, affinity CPUs 0,1; Rust 1.88.0, FFmpeg 7.1.5, Mesa 25.0.7 Lavapipe with two rasterizer threads. Root release uses opt-level=s/thin LTO/one codegen unit. editor3 overrides only cap-editor to level 3; it is not a full GPUI shipping-profile or physical-GPU measurement. Source video is generated 320×180 H.264 at 1 fps; output is 640×360 at 15 fps. Microphone inputs are stereo 48 kHz PCM16 WAV or AAC-LC/MP4 at 96 kb/s, generated from deterministic tones. There is no camera/cursor drawing, music or caption track.

Baseline 0983901353c759f4857520afdfdc5fe12b2ff19a. Raw repeated samples, saved binaries, comparison JSON and hashes are retained locally outside the production diff.

Correctness and bounds

  • A new differential test reproduces Invalid sequential export audio window on the baseline, then compares complete sample bits against the existing in-memory renderer after repeated short cuts. It exercises all isolation tiers, mono/stereo microphones, positive/negative offsets, source switches, one-sample and non-aligned requests, and source exhaustion.
  • A direct history test verifies read-back uses already-decoded samples without advancing the decoder, rejects reads before stored history, checks complete stored samples against full decoding, and enforces a fixed length bound while the window advances.
  • Ordinary reads retain the existing zero-history behavior. Enhanced samples retain the existing maximum request window plus 28,320 history frames, independent of recording duration. Vec capacity and other decoder/model allocations are separate from this sample-length bound.
  • Existing eligibility tests still reject backward/unsupported timelines. Streaming eligibility now accepts short forward cuts and exact splits.

Validation

  • 24 export-audio tests pass; two preexisting real-recording tests require local fixtures and remain ignored.
  • 47 cap-export library tests pass.
  • Three history-filtered release tests pass at both size and editor-only level-3 settings.
  • cargo fmt --all --check and editor/export all-target checks pass.
  • Strict editor all-target Clippy and strict cap-export library/new-example Clippy pass with --no-deps.
  • Strict cap-export all-target Clippy is blocked by the unchanged Linux-only irrefutable let/else in mp4.rs:2120; the same code is present on the baseline. No lint suppression or unrelated cleanup is included.

Reproduction

Create a fresh fixture directory; the example writes its project config/metadata. Copy the new example onto the baseline, build it on both revisions and save the binaries separately:

cargo build -p cap-export --release --locked --example studio-sound-cut-benchmark
cargo build -p cap-export --release --locked --example studio-sound-cut-benchmark \
  --config profile.release.package.cap-editor.opt-level=3
mkdir -p /tmp/cap-short-cut
ffmpeg -f lavfi -i testsrc2=size=320x180:rate=1 -t 3600 \
  -c:v libx264 -preset ultrafast -threads 2 -g 1 -pix_fmt yuv420p \
  /tmp/cap-short-cut/display.mp4
ffmpeg -f lavfi \
  -i 'aevalsrc=0.14*sin(2*PI*180*t)+0.04*sin(2*PI*817*t)|0.11*sin(2*PI*240*t)+0.03*sin(2*PI*1103*t):s=48000' \
  -t 3600 -c:a pcm_s16le /tmp/cap-short-cut/mic.wav
taskset -c 0,1 /path/to/saved/studio-sound-cut-benchmark \
  /tmp/cap-short-cut /tmp/cap-short-cut/result.mp4

For AAC, encode the generated WAV to mic.m4a with -c:a aac -b:a 96k, then expose that file as the fixture's mic.wav path (for example, with a symlink in a separate fixture directory). FFmpeg detects the container from its contents. The saved benchmark binaries are identical for both source formats. Set CAP_EXPORT_CUT_GAP=0 or 1 for the already-streaming controls.

Use the contributor guide's native environment. To reproduce the headless measurements, set VK_DRIVER_FILES to a Lavapipe ICD, CAP_RENDER_FORCE_SOFTWARE_ADAPTER=1, LP_NUM_THREADS=2, and a writable XDG_RUNTIME_DIR. Alternate five baseline/head processes. The example emits preparation/first-progress/total timings and Linux VmHWM. Compare decoded video (-map 0:v:0 -pix_fmt rgba) and PCM (-map 0:a:0 -c:a pcm_f32le) using FFmpeg's SHA-256 hash muxer, outside timing.

Risk is confined to retaining and addressing a bounded decoded-sample prefix, and enabling the existing streaming pipeline for short forward cuts. There is no new public API, dependency, unsafe code, persistent cache, neural model or audio-format change.

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge; no actionable issues were found.

Summary

Enables streaming audio for short forward cuts with Studio Sound by retaining a bounded prefix of decoded samples.

  • Studio Sound exports stream through short forward cuts.

No actionable issues found. Review used code inspection; tests and benchmark measurements were not rerun.

Reviews (1) · Last reviewed commit: "perf(editor): stream short Studio Sound ..." · Reviewed by Greptile

@vercel

vercel Bot commented Oct 25, 2026

Copy link
Copy Markdown
Contributor

Someone is attempting to deploy a commit to the Cap Software Inc Team on Vercel.

A member of the Team first needs to authorize it.

@richiemcilroy
richiemcilroy merged commit af5820f into CapSoftware:main Oct 25, 2026
48 of 50 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants