Skip to content

perf(media-server): parallelise recording verification behind opt-in env settings - #3009

Merged
richiemcilroy merged 3 commits into
mainfrom
perf/media-server-verify-experiments
Oct 23, 2026
Merged

richiemcilroy merged 3 commits into
mainfrom
perf/media-server-verify-experiments

Conversation

@richiemcilroy

@richiemcilroy richiemcilroy commented Oct 23, 2026 •

Copy link
Copy Markdown
Member

Desktop recordings spend most of their media-server time in verification. On an 8 vCPU worker a 5-minute recording takes ~30s to verify, and almost all of that is one H.264 decode that stops scaling at ~6 cores. This makes verification parallel without changing what it checks. Everything new is behind env settings that default to today's behaviour.

What changes

  • Packet proof reads run concurrently (always on). The six independent reads of the source and output tracks used to run one after another. They now start together, and their results are still checked in the original order with the same errors. Leftover reads are aborted on the first failure. On a 5-minute recording this takes the proof from 3.6s to about 1.5s.
  • Parallel decode of packet-proven recordings (MEDIA_SERVER_VERIFY_DECODE_CHUNKS, off by default).
    • The video is decoded as N runs of whole source fragments in parallel, using the same strict flags (-xerror, -err_detect explode+crccheck).
    • Their frame lines are replayed in order, with offsets, through the existing parser and validateEvidence.
    • Audio is still decoded from the output file.
    • This only applies after the packet proof has shown the output's packets are byte-identical to the source's.
  • Split points are proven, not assumed. A fragment can open with a sync sample that the decoder still can't start cleanly from; on one real recording, a run starting at such a fragment loses one frame with no error. So a fragment only becomes a split point if decoding it alone gives a frame for every packet. The total decoded frame count must also match the packet count. Any problem in the parallel path, whether a decoder error, a frame-count gap, odd stream parameters or no clean split point, falls back to the existing single decode. So the parallel path can only say "verified"; a corrupt recording is still judged by the single decode.
  • Parallel re-read of the uploaded output (MEDIA_SERVER_VERIFY_DOWNLOAD_CONCURRENCY, off by default).
    • It's the same proof as before: every byte is hashed in order.
    • Every range is pinned with If-Match and its ETag and Content-Range are checked.
    • The retry rules and error messages are the same.
    • Storage that ignores Range falls back to the sequential read.

Measured. Test boxes with the desktop segment fixtures, 2 runs each; total server time per mux job:

1 min 5 min 20 min
today, 8 vCPU (prod workers) 10.6s 39.1s 141.0s
this PR, 8 vCPU, 4 chunks 8.3s 26.2s 93.0s
this PR, 16 vCPU, 4 chunks 6.6s 19.6s 67.1s
this PR, 32 vCPU, 8 chunks 5.7s 16.0s 52.2s

The muxed MP4, thumbnail and GIF are byte-identical to today's build in every configuration.

Rejected along the way (not in this PR):

  • GPU (NVDEC) verification: slower, and it passed all four corrupted test files that the CPU decode rejects.
  • More job slots per worker: no throughput gain, because the CPU is saturated.

Tests

  • 3 chunked tests:
    • evidence deep-equals the single decode;
    • fragments that don't start on keyframes fall back with identical evidence;
    • a corrupt fragment is still rejected.
  • 5 ranged re-read tests:
    • correct hash and ranges;
    • a range from a replaced object is rejected;
    • a wrong Content-Range is rejected;
    • transient 503 retry;
    • fallback when ranges are ignored.
  • The existing verification suites pass unchanged.

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge; no new blocking issues were found.

Summary

This PR speeds up recording verification while keeping parallel decoding and downloads behind opt-in settings.

  • Packet proof reads run at the same time.
  • Recording checks can decode proven video fragments in parallel.
  • Large upload checks can hash byte ranges in parallel.

Reviews (2) · Last reviewed commit: "fix(media-server): stop chunk copies on ..." · Reviewed by Greptile

@vercel
vercel Bot temporarily deployed to Preview – cap-status October 23, 2026 18:29 Inactive
@vercel

vercel Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
cap-status Skipped Skipped Oct 23, 2026 6:52pm UTC

@richiemcilroy

Copy link
Copy Markdown
Member Author

@greptileai review

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found 2 potential issues (2 medium).

Open in Capy

Comment thread apps/media-server/src/lib/recording-verification.ts Outdated
Comment thread apps/media-server/src/lib/recording-verification.ts Outdated
@vercel
vercel Bot temporarily deployed to Preview – cap-status October 23, 2026 18:34 Inactive
Comment thread apps/media-server/src/lib/recording-verification.ts Outdated
Comment thread apps/media-server/src/lib/recording-verification.ts Outdated
Comment thread apps/media-server/src/__tests__/lib/recording-verification.integration.test.ts Outdated

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

View 1 other finding in Capy.

Open in Capy

@vercel
vercel Bot temporarily deployed to Preview – cap-status October 23, 2026 18:52 Inactive
@richiemcilroy

Copy link
Copy Markdown
Member Author

@greptileai review

@richiemcilroy
richiemcilroy merged commit e770f98 into main Oct 23, 2026
43 checks passed

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

View 1 other finding in Capy.

Open in Capy

This branch was successfully deployed

1 active and 1 inactive deployments
Preview – cap-web — 237b874d Deployed Oct 23, 2026 by vercel[bot]
Preview – cap-status — 237b874d Deployed Oct 23, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant