feat(mmmu-pro): mirror media to a dedicated bucket with shared mirror plumbing

[ayush-or]

TL;DR

MMMU Pro's Hugging Face image URLs expire after one hour, while cached dataset pages last 24 hours or indefinitely when revision-pinned. Use the VGI-Bench mirror pattern (#37, #41, #42): publish the original media once to a dedicated mmmu-pro-mirror R2 bucket and resolve samples through a committed stable-URL manifest. This replaces the closed base64 approach in #86.

The mirror has been run against the dedicated mmmu-pro-mirror bucket (public origin https://mmmu-pro-mirror.openrouter.ai): 1,730/1,730 images, 1.39 GB, every object verified by public readback, and the resulting src/benchmarks/mmmu-pro-media-manifest.json (837 KiB) is committed and registered. Default mmmu_pro_vision runs now send mirror URLs instead of signed HF URLs.

What changed?

• scripts/media-mirror.ts now holds everything both mirror scripts share: env parsing, HF /rows paging with 429/5xx retry, download, SHA-256, uploadUnlessPresent (skip when the stored object size matches, --force overrides, --dry-run writes nothing), verifyPublished (public GET, content type, SHA-256), bounded concurrency, and hashManifestEntries. mirror-vgi-bench-media.ts is refactored onto it with unchanged behavior; its committed manifestHash reproduces byte-for-byte with the shared hasher.
• mirror-mmmu-pro-media.ts gains skip-existing, --force, --dry-run, and defaults --revision to MMMU_PRO_DEFAULT_REVISION. It keeps the stricter checks: 40-hex revision, x-revision header, revision-pinned cached-asset paths, non-empty bytes, exactly 1,730 rows, public readback of every object, and no manifest written on any failure.
• Runtime: mmmu-pro-media-manifest.ts imports the committed JSON at build time, validates it with buildMmmuProMediaManifest, and exposes mmmuProMediaManifestFor(revision). makeMmmuProVisionDatasetLayer pins the HF revision and, like makeVgiBenchDatasetLayer, swaps in the mirror URL and stamps media_manifest_hash / dataset_revision when a committed manifest matches. mmmu_pro_vision configs accept an optional datasetRevision (same as vgi-bench); any revision without a committed manifest falls back to signed HF URLs.
• readMediaMirrorEnv strips a trailing /<bucket> from BENCH_MEDIA_S3_ENDPOINT. The R2 dashboard displays the S3 API URL in that form, and pasting it verbatim made Bun's S3Client write to <bucket>/<key> inside the bucket (stat succeeded, public GET 404).
• scripts/README.md documents the per-benchmark bucket layout, Cloudflare provisioning steps, BENCH_MEDIA_* variables, and how to run both scripts.

HF /rows ignores revision=. Preparation checks the actual x-revision header, and runtime source-path checks prevent silently pairing a changed dataset row with the mirrored image from another revision.

Why?

Same shape as vgi-bench so the two mirrors are maintained together, with a separate bucket per benchmark so licensing, takedowns, and write tokens stay isolated. MMMU Pro is a fixed 1,730-row set with one image each, so unlike vgi-bench there is no reason to tolerate unresolved assets.

How to test

Run bun run format:check, bun run check, bun run typecheck, bun test, and bun run build. All passed at 376b5d4, 1,468 tests. Lint has only pre-existing warnings.

Manual checks:

• bun run mirror-mmmu-pro-media with the mmmu-pro-mirror R2 token uploaded and public-readback-verified 1,730/1,730 objects for 563f3e84bb3b90893083a1f039cfa13077f2302b and wrote the committed manifest. Re-running it is a verification pass (all objects skip as size-matched).
• Manifest sanity: 1,730 unique ids and sha256s, all URLs under https://mmmu-pro-mirror.openrouter.ai/mmmu-pro/563f3e84.../, all sourcePaths under the pinned cached-asset prefix, only image/png and image/jpeg, no signed-URL query params. Spot-check HEAD on four mirror URLs returned 200 with matching content type and length.
• New unit test asserts the committed manifest is selected for MMMU_PRO_DEFAULT_REVISION with 1,730 entries on the mirror origin, and that an unknown revision yields no manifest.
• vgi-bench manifestHash recomputed with hashManifestEntries equals the committed value.

Not yet done: a provider-fetch smoke run (a real mmmu_pro_vision sample through a provider to confirm it fetches from the mirror origin).

Benchmark impact

Source: MMMU/MMMU_Pro, vision/test, revision 563f3e84bb3b90893083a1f039cfa13077f2302b; declared Apache-2.0. Images are copied without resizing or re-encoding. Prompts, answers, and scoring stay unchanged. Default runs now pin that revision (previously HEAD), which also makes the HF row cache non-expiring. Mirrored runs fail explicitly on missing media or source drift.

Reviewer focus

• scripts/media-mirror.ts and its tests are the shared surface; the vgi diff should be pure extraction.
• src/benchmarks/mmmu-pro-media-manifest.json is generated output; review a few entries rather than the whole file.
• A provider-fetch smoke run (one real mmmu_pro_vision sample through a provider) has not been done yet; the mirror URLs are verified by direct public GET only.

Checklist

• Tests cover changed behavior
• Public API or configuration changes are backward compatible, or the break is documented
• Benchmark changes document dataset provenance and licensing
• No credentials, private results, or restricted dataset contents are included
• Documentation is updated where needed

Link to Devin session: https://openrouter.devinenterprise.com/sessions/3544413f10474e33b49c166fe6a5db24
Open in Devin Desktop: https://openrouter.devinenterprise.com/desktop/session/3544413f10474e33b49c166fe6a5db24?variant=devin
Requested by: @ayush-or