[Benchmark] Add MET-Bench (ICML 2026, entity-state tracking)

[vanyacohen]

Summary

• Adds MET-Bench (paper · project page · 🤗 datasets: Minecraft, Chess, Shell Game), an ICML 2026 benchmark for entity-state tracking across three subdomains: Minecraft (multiple-choice next-state prediction), Chess (final board-state prediction), and Shell Game (final hidden-ball location). Each subdomain includes parallel text and image tasks.
• State-of-the-art VLMs generally perform worse on the image tasks than on the equivalent text tasks, even with explicit reasoning. Adding MET-Bench makes this gap in visual state tracking directly measurable through VLMEvalKit.

Implementation

Registers six datasets through METBenchImage and METBenchText, with 500 evaluation examples per task, benchmark prompts and scoring, tests, and usage documentation linked from the documentation index.

Usage

python run.py \
  --model YOUR_CONFIGURED_MODEL \
  --data METBench_minecraft_text METBench_minecraft_image \
         METBench_chess_text METBench_chess_image \
         METBench_shell_text METBench_shell_image \
  --work-dir results/metbench

See MET-Bench usage for configuration and a small evaluation example.

Validation

• Native GPT-4o-mini evaluation: 500 examples per task, 3,000 completed responses. Independently re-scored saved predictions reproduce every task result.
• Seven MET-Bench tests passed on Linux, covering registration, data loading, prompt ordering, scoring, and confidence intervals; the combined dataset and inference test run passed all 36 tests.
• Qwen2.5-VL-3B smoke evaluation completed all six tasks.
• pre-commit run --all-files: passed.

Subdomain	Text accuracy (%)	Image accuracy (%)
Minecraft	64.20	31.60
Chess	68.82	49.88
Shell Game	63.60	30.60

Compatibility

• Adds chess>=1.11.2,<2 for board-state scoring.
• Image tasks require a model supporting multiple interleaved images.

[TianhaoLiang2000]

Thanks for your contribution!

Since MET-Bench consists of multiple implementation files, could you please organize it as a dedicated package under vlmeval/dataset/metbench/? The benchmark-specific README/documentation, license, and requirements can also be placed in this directory.

In particular, chess is only required by MET-Bench, so it can be listed in vlmeval/dataset/metbench/requirements.txt, with the installation command documented in the README. We prefer to avoid adding benchmark-specific documentation and dependencies to the global docs index and requirements.txt. The registration changes in vlmeval/dataset/init.py can remain.