[Model][Dataset] Add GRT profiles and DIVE-Bench educational/high-motion tasks

[Hai-chao-Zhang]

High-Motion release hold — 14 September 2026

All High-Motion quality rankings are withheld pending target/reference consistency review. A source-bound check of four canonical references found trajectories matching a left-index joint projection while the questions request a right-hand palm/ring-base target. A subsequent fixed-formula audit covered all 3,243 references (826,253 frames): 2,871 complete trajectories match that fixed projection within the predeclared tolerance, while 3,121 references have all grid labels match. Neither check matches every item. This does not establish universal incorrect labels, recover the original constructor, or establish intended anatomy. No model predictions or fitted mapping were used.

No High-Motion GRT superiority claim is supported. The released GRT adapters now explicitly accept only dive_bench_educational_high_fps and densevideo; all High-Motion aliases and other tasks are rejected. Raw High-Motion tasks and ordinary compatible models remain available for diagnostics, not promotion to the current leaderboard. No labels, sampling rules, model kernels, profiles, or thresholds were changed to obtain an apparent win.

The completed native five-arm, two-item High-Motion GPU smoke establishes execution/input alignment only, not full benchmark correctness or quality advantage. Fresh full Educational GPU/judge reproduction remains queued. This PR remains a draft pending resolution and access/reproduction review. See the source-audit follow-up and release criteria.

Summary

• Add DIVE-Bench's Educational High-FPS Videos and High-Motion High-FPS Videos tasks, with separate full/preview identifiers and compatibility aliases.
• Add optional, isolated GRT adapters for the three educational profiles in the public leaderboard. The bridge calls the actual pinned GRT runtime; stock model implementations remain unchanged.
• Draft for integration review: CPU/protocol checks pass, but public dataset access and a fresh full GPU inference/judge reproduction are not yet validated.

In scope

• Five native task IDs: dive_bench_educational_high_fps (634 QA rows / 317 videos), dive_bench_high_motion_high_fps (3243 clips), dive_bench_high_motion_high_fps_preview1000 (first 1000 rows), densevideo, and densevideo_highmotion (preview alias).
• Exact annotation-file SHA-256, pinned revisions for both sources, required schema/counts, unique (video_path, qid) identities, exact stored references, complete prediction coverage and safe video-path resolution preserving repeated numeric names across action directories. Both reviewed high-motion Parquet serializations also pass an independent full ordered-content fingerprint before preview selection.
• Endpoint-inclusive High-Motion sampling and temporal selection of stored references; the body-part semantics of those references remain under the hold above. Objective Educational CER/WER/Token-F1/Exact Match and High-Motion grid accuracy/ADE/FDE/transition accuracy/Token-F1 are preserved. Disabled judge calls do not create zero MOS scores.
• Atomic annotation-table publication under a lock; missing videos, unsupported video-only wrappers, mismatched decoded frame counts, incomplete predictions and altered release bytes fail explicitly.
• Lazy GRT profile bridge, real six-argument generate_until requests without reference-answer leakage, exact generation caps, serial one-GPU execution and the historical effective seed tuple. Baseline/all/candidate roles remain distinct.

Validation

• pytest -q tests/test_dive_bench.py tests/test_grt.py: 150 CPU tests, including a real synthetic MP4 decoded by Decord and compared pixel-for-pixel against endpoint-inclusive frames; native registration, annotation handling, objective metrics, atomic cache writes, prompt alignment, seed settings and GRT request/state-restoration contracts. Offline tests assert the production source-pin/checksum constants and exercise byte/content tampering, serialization compatibility, row reordering and out-of-preview changes using synthetic full-size fixtures with fixture-specific expected hashes. These fixtures are not fresh downloads of canonical references. Tests also cover distributed/MPI/Slurm metadata rejection before CUDA queries and Educational-only scope guards across all three profiles and three roles. Unsupported task/profile binding fails before optional model loading, while unsupported tasks are rejected before prompt construction or generation.
• The repository's exact pre-commit run --all-files workflow passes locally, including pinned flake8/isort/YAPF/hooks; git diff --check passes. The earlier fork lint failure was reproduced and corrected with import formatting in three integration files, without a repository-wide formatting change.
• Remote CI at this head: owned-fork lint passed. Upstream lint and PR tests require maintainer workflow approval (action_required); they have not passed or failed code execution.
• Eleven objective metric/parser functions match the audited release's ASTs. Educational suffix and high-motion prompts are compared against the installed minimal runtime.
• Editable VLMEvalKit installation is the supported upstream route. An earlier standalone-wheel audit found pre-existing omitted MEGABench modules preventing normal import; this PR does not claim standalone-wheel support or modify unrelated upstream packaging.

These checks do not download model weights, execute GPU inference, measure benchmark scores, establish statistical significance, or validate access to restricted videos.

Access and compatibility

• Educational annotations are gated. On 2026-09-14 the owner verified that the high-motion source is private at pinned revision d44407f607fdf020c59b816884f06ed6d453cf26. Its actual Hub Parquet SHA-256 is 518e2896749b4d6e957d7e9fb0ae16f75c28954e50ef84303889070253cf8ecd; the recorded legacy serialization has identical ordered task content. Owner access does not establish public access. No dataset/model assets, answers, credentials or videos are redistributed by this PR. Underlying LPM/YouTube and EgoDex terms are not replaced by the code license.
• Separately, the owner configured a private haichaozhang/DIVE-Bench entry at d80461fccf879d5efdeece0edce8608a72d64f10 with exactly educational_high_fps (634) and high_motion_high_fps (3243), and corresponding named configs on the old sources. All four owner-authorized fresh-cache annotation loads passed; original video archives, data pins, source visibility and gating are unchanged. This integration retains its original pinned source repositories.
• Raw High-Motion diagnostic evaluation supplies sampled images and rejects VIDEO_LLM=True wrappers that would resample or dispatch to video-only generation. Use an ordinary compatible ordered-image model; the released GRT bridge explicitly rejects High-Motion. Compatibility with every multi-image wrapper is not claimed.
• The optional GRT runtime owns the lmms_eval namespace: use a dedicated environment, not an existing separate lmms-eval installation. Its runtime dependency intentionally remains pinned to a52c0360de0df175a476e815f4aaa27886131b23, matching the frozen GPU reproduction source; this is distinct from the newer public report/export-guard release below. Model kernels, thresholds and generation settings are unchanged. Runtime commands are in docs/en/DIVE-Bench.md.
• The three released GRT profiles are Educational-only, including their matched baseline/all-patch/candidate roles. The historical High-Motion GRT row used a different legacy wrapper and incompatible input protocols, so it cannot establish a matched advantage. All High-Motion results—not just earlier protocol exclusions—are currently withheld; no ranked or unranked High-Motion numeric table or current CSV row is published. Original audit evidence remains unchanged.
• The current public minimal release 9ee16af0d03d7f31e726b71f00e4586972afb062 generates 29 Educational result rows plus 12 controls (41 CSV records), with zero High-Motion result rows under the target/reference hold. The immutable 32-row snapshot and 27-run historical audit remain preserved. Release CPU validation passed (343 tests plus 7 subtests). Separate packaging checks also passed: an isolated sdist-to-wheel build and actual installed-wheel CLI execution outside the checkout. All three public CI jobs passed. Historical numerical reconstruction is not fresh full GPU inference, and a positive historical point estimate does not establish statistical significance.
• The default is eight sampled frames, not dense processing of high-FPS source videos. Patch-projection ratios are not end-to-end FLOP reductions. Open MOS requires the separate pinned judge/matched-control workflow.
• Task names follow the revised manuscript bundled with the public code. The earlier arXiv version is educational-only; counts, judge definitions and sampling claims need to be reconciled before a final paper release.

Public code and score audit: https://github.com/Hai-chao-Zhang/DenseVideoUnderstand/tree/release/dive-bench-minimal

Project: https://www.zhanghaichao.xyz/DenseVideoUnderstand/

Type of change

• New benchmark
• New model integration
• Tests and documentation
• Stock model behavior change