[Benchmark] Add MET-Bench (ICML 2026, entity-state tracking)

docs/en/METBench.md

@@ -0,0 +1,42 @@
+# MET-Bench
+
+[Paper](https://arxiv.org/abs/2502.10886) · [Website](https://vanyacohen.com/MET-Bench/) · [Reference evaluator](https://github.com/vanyacohen/MET-Bench)
+
+MET-Bench evaluates entity tracking in Minecraft, Chess, and Shell Game through parallel text and image inputs. Chess and Shell Game track ten actions from an initial state; Minecraft predicts the next state after an action.
+
+## Datasets
+
+| Domain | Hugging Face dataset | Text task | Image task |
+|---|---|---|---|
+| Minecraft | [🤗 Minecraft](https://huggingface.co/datasets/vanyacohen/MET-Bench-Minecraft) | `METBench_minecraft_text` | `METBench_minecraft_image` |
+| Chess | [🤗 Chess](https://huggingface.co/datasets/vanyacohen/MET-Bench-Chess) | `METBench_chess_text` | `METBench_chess_image` |
+| Shell Game | [🤗 Shell Game](https://huggingface.co/datasets/vanyacohen/MET-Bench-Shell) | `METBench_shell_text` | `METBench_shell_image` |
+
+Each domain contains 500 unique test inputs. Dataset revisions are pinned, and the text and image tasks use the same ordered example IDs. Text tasks download `evaluation_text_only`; image tasks download `evaluation`. Chess and Shell Game are deduplicated by initial state and ten-action prefix. Minecraft is deduplicated by initial state, action, and ordered candidate states.
+
+## Run
+
+After installing VLMEvalKit, select a model that supports multiple interleaved images:
+
+```bash
+python run.py \
+  --model YOUR_CONFIGURED_MODEL \
+  --data METBench_minecraft_text METBench_minecraft_image \
+         METBench_chess_text METBench_chess_image \
+         METBench_shell_text METBench_shell_image \
+  --work-dir results/metbench
+```
+
+The dataset supplies the benchmark's chain-of-thought prompts. Set temperature to zero and the output limit to 4,096 tokens in the model configuration for the reference evaluation. Credentials are read from the environment.
+
+For a small check, select two examples:
+
+```bash
+python run.py \
+  --model YOUR_CONFIGURED_MODEL \
+  --data METBench_minecraft_image \
+  --data-config '{"METBench_minecraft_image":{"class":"METBenchImage","dataset":"METBench_minecraft_image","limit":2}}' \
+  --work-dir results/metbench-check
+```
+
+Each task produces an `_acc.csv` file. `Overall` reports accuracy in percent: correct answer choices for Minecraft and Shell Game, and correctly predicted board squares for Chess. `ci_lower` and `ci_upper` give 95% confidence bounds: a normal approximation using the standard error across board scores for Chess, and Wilson intervals across answer choices for Minecraft and Shell Game. Chess bounds are unavailable for a single example; `examples` gives the evaluated example count. Prompts and answer parsing match the reference evaluator and the lmms-eval integration.

docs/en/index.rst

@@ -26,6 +26,7 @@ We always welcome users' PRs (Pull Requests) and Issues to improve VLMEvalKit!
 
    Development.md
    ConfigSystem.md
+   METBench.md
 
 .. _Other Notes:
 .. toctree::

requirements.txt

@@ -5,6 +5,7 @@ apted>=1.0.3
 bert_score
 cairosvg
 cd-fvd
+chess>=1.11.2,<2
 colormath>=3.0.0
 datasets
 decord>=0.6.0

tests/test_metbench.py

@@ -0,0 +1,117 @@
+"""MET-Bench's native dataset, prompt ordering, and result-file contracts."""
+
+import io
+
+import pandas as pd
+import pytest
+from datasets import Dataset
+from PIL import Image
+
+from vlmeval.dataset import SUPPORTED_DATASETS, build_dataset
+from vlmeval.dataset.metbench import EVALUATION_RELEASES, METBenchImage, METBenchText
+from vlmeval.smp import dump
+from vlmeval.smp.file import INFER_FAIL_MSG
+
+FEN = 'rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1'
+
+
+def image(index):
+    """Encode distinguishable frames to detect reordering and action-frame leakage."""
+    buffer = io.BytesIO()
+    Image.new('RGB', (2, 2), (index, 0, 0)).save(buffer, format='PNG')
+    return {'bytes': buffer.getvalue(), 'path': None}
+
+
+def rows(domain, modality):
+    """Make a full-size synthetic split with sparse source indices."""
+    if domain == 'minecraft':
+        row = dict(initial_state='{"x":0}', action='Walk forward 1 block.',
+                   candidate_states=['{"x":1}', '{"x":2}', '{"x":3}', '{"x":4}'], correct_choice=2)
+        if modality == 'image':
+            row.update(image_initial_state=image(0), image_action=image(99),
+                       image_candidate_states=[image(i) for i in range(1, 5)])
+    else:
+        row = dict(initial_state=FEN if domain == 'chess' else 1,
+                   final_state=FEN if domain == 'chess' else 2,
+                   actions=['g1f3' if domain == 'chess' else '1 swap 2'] * 10)
+        if modality == 'image':
+            row['image_actions'] = [image(i) for i in range(10)]
+    return Dataset.from_list([dict(row, example_id=f'{domain}-test-{3 * i}') for i in range(500)])
+
+
+@pytest.mark.parametrize('domain', ['minecraft', 'chess', 'shell'])
+@pytest.mark.parametrize('modality', ['text', 'image'])
+def test_native_dataset_contract(monkeypatch, tmp_path, domain, modality):
+    """Check discovery, pinned downloads, prompts, and scoring through native APIs."""
+    calls = []
+
+    def load(repo, config, **kwargs):
+        calls.append((repo, config, kwargs))
+        return rows(domain, modality)
+
+    monkeypatch.setattr('vlmeval.dataset.metbench.load_dataset', load)
+    monkeypatch.setattr('vlmeval.dataset.metbench.LMUDataRoot', lambda: str(tmp_path))
+    name = f'METBench_{domain}_{modality}'
+    assert name in SUPPORTED_DATASETS
+    registered = build_dataset(name)
+    assert len(registered.data) == 500
+    cls = METBenchText if modality == 'text' else METBenchImage
+    task = cls(name, limit=2)
+    repo, revision = EVALUATION_RELEASES[domain]
+    assert calls[-1] == (repo, 'evaluation_text_only' if modality == 'text' else 'evaluation',
+                         {'split': 'test', 'revision': revision})
+    assert task.data['index'].tolist() == [0, 3]
+    prompt = task.build_prompt(task.data.iloc[1])
+    images = [part['value'] for part in prompt if part['type'] == 'image']
+    expected = list(range(5 if domain == 'minecraft' else 10)) if modality == 'image' else []
+    assert [Image.open(path).getpixel((0, 0))[0] for path in images] == expected
+    assert task.dump_image(task.data.iloc[1]) == images
+    if modality == 'text':
+        assert len(prompt) == 1
+    original_text = [part['value'] for part in prompt if part['type'] == 'text']
+    task.by_index['3']['target'] = 'PRIVATE_TARGET'
+    assert [p['value'] for p in task.build_prompt(task.data.iloc[1]) if p['type'] == 'text'] == original_text
+    task.by_index['3']['target'] = task.examples[0]['target']
+    path = str(tmp_path / 'predictions.xlsx')
+    predictions = task.data.copy()
+    predictions['prediction'] = [f'FINAL ANSWER: {task.examples[0]["target"]}', 'unparseable']
+    dump(predictions.iloc[::-1], path)
+    result = task.evaluate(path).iloc[0]
+    assert result['Overall'] == 50.0
+    assert result['ci_lower'] < 50 < result['ci_upper']
+    assert result['examples'] == 2
+    predictions['prediction'] = [None, 'unparseable']
+    dump(predictions, path)
+    assert task.evaluate(path).iloc[0]['Overall'] == 0
+    predictions['prediction'] = INFER_FAIL_MSG
+    dump(predictions, path)
+    with pytest.raises(ValueError, match='failed inference'):
+        task.evaluate(path)
+    dump(pd.concat([predictions.iloc[:1]] * 2), path)
+    with pytest.raises(ValueError, match='duplicate'):
+        task.evaluate(path)
+    dump(predictions.iloc[:1], path)
+    with pytest.raises(ValueError, match='selected examples'):
+        task.evaluate(path)
+
+
+def test_chess_confidence_uses_whole_trials(monkeypatch, tmp_path):
+    """Correlated squares cannot inflate the effective example count."""
+    import math
+
+    monkeypatch.setattr('vlmeval.dataset.metbench.load_dataset', lambda *a, **k: rows('chess', 'text'))
+    monkeypatch.setattr('vlmeval.dataset.metbench.LMUDataRoot', lambda: str(tmp_path))
+    task = METBenchText('METBench_chess_text', limit=100)
+    predictions = task.data.copy()
+    predictions['prediction'] = [f'FINAL ANSWER: {FEN}', 'unparseable'] * 50
+    path = str(tmp_path / 'chess.xlsx')
+    dump(predictions, path)
+    result = task.evaluate(path).iloc[0]
+    margin = 100 * 1.959963984540054 * math.sqrt(0.25 / 99)
+    assert result['Overall'] == 50.0
+    assert result['ci_lower'] == pytest.approx(50 - margin)
+    assert result['ci_upper'] == pytest.approx(50 + margin)
+    task = METBenchText('METBench_chess_text', limit=1)
+    dump(predictions.iloc[:1], path)
+    result = task.evaluate(path).iloc[0]
+    assert pd.isna(result['ci_lower']) and pd.isna(result['ci_upper'])

vlmeval/dataset/METBENCH_LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 vanyacohen
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

vlmeval/dataset/__init__.py

@@ -79,6 +79,7 @@
 # Add by EASI team
 from .megabench import MEGABench
 from .memlens import MemLens
+from .metbench import METBenchImage, METBenchText
 from .miabench import MIABench
 from .mindcubebench import MindCubeBench
 from .mlvu import MLVU, MLVU_MCQ, MLVU_OpenEnded
@@ -294,6 +295,7 @@ def evaluate(self, eval_file, **judge_kwargs):
 
 # Add new supported dataset class here
 IMAGE_DATASET = [
+    METBenchImage,
     ImageCaptionDataset, ImageYORNDataset, ImageMCQDataset, ImageVQADataset,
     MathVision, LENS, MMMUDataset, OCRBench, MathVista, LLaVABench, LLaVABench_KO, VGRPBench, MMVet,  # noqa: E501
     MTVQADataset, TableVQABench, MMLongBench, MemLens, MMLongBenchDoc, VCRDataset, MMDUDataset, DUDE, DocScope,
@@ -350,6 +352,7 @@ def evaluate(self, eval_file, **judge_kwargs):
 VIDEO_DATASET += [SiteBenchVideo, VsiBench, VsiSuperRecall, VsiSuperCount, MMSIVideoBench, STIBench, DSRBench]  # noqa: E501
 
 TEXT_DATASET = [
+    METBenchText,
     TextMCQDataset, SGI_Bench_Wet_Experiment, SGI_Bench_Dry_Experiment,
     SGI_Bench_Deep_Research, SGI_Bench_Idea_Generation, XSTestDataset, FlamesDataset,
     MedXpertQAText

vlmeval/dataset/metbench.py

@@ -0,0 +1,164 @@
+"""MET-Bench datasets for VLMEvalKit."""
+
+import hashlib
+import io
+import math
+from pathlib import Path
+
+import pandas as pd
+from datasets import Image as DatasetImage
+from datasets import List, load_dataset
+
+from vlmeval.dataset.image_base import ImageBaseDataset
+from vlmeval.dataset.metbench_core import build_messages, clustered_ratio_interval, score
+from vlmeval.smp import LMUDataRoot, dump, load
+from vlmeval.smp.file import INFER_FAIL_MSG
+
+EVALUATION_RELEASES = {
+    "minecraft": (
+        "vanyacohen/MET-Bench-Minecraft",
+        "ae0b474dd22986b8b292c2a42f5e81cf8675661e",
+    ),
+    "chess": ("vanyacohen/MET-Bench-Chess", "6fd525cd537a9c25efe64ae88102c0ab68f94b1d"),
+    "shell": ("vanyacohen/MET-Bench-Shell", "75725717b63cf6e54a5b8abd22e6ad96b47a12e3"),
+}
+
+
+class METBenchImage(ImageBaseDataset):
+    """Evaluate paired MET-Bench tasks with the released prompts and metrics."""
+
+    TYPE = "VQA"
+    MODALITY = "IMAGE"
+    modality = "image"
+    force_use_dataset_prompt = True
+
+    @classmethod
+    def supported_datasets(cls):
+        return [
+            f"METBench_{domain}_{cls.modality}"
+            for domain in ("minecraft", "chess", "shell")
+        ]
+
+    def __init__(self, dataset="METBench_minecraft_image", limit=500):
+        if dataset not in self.supported_datasets():
+            raise ValueError(f"Unsupported MET-Bench task: {dataset}")
+        if not isinstance(limit, int) or not 1 <= limit <= 500:
+            raise ValueError("limit must be between 1 and 500")
+        self.dataset_name = dataset
+        self.domain = dataset.split("_")[1]
+        self.img_root = str(Path(LMUDataRoot()) / "images" / "METBench")
+        self.meta_only = True
+        self.skip_noimg = False
+        repo, revision = EVALUATION_RELEASES[self.domain]
+        config = "evaluation_text_only" if self.modality == "text" else "evaluation"
+        data = load_dataset(repo, config, split="test", revision=revision)
+        if len(data) != 500:
+            raise ValueError("Expected the released 500-example evaluation split")
+        if self.modality == "image":
+            if self.domain == "minecraft":
+                columns = {
+                    "image_initial_state": DatasetImage(decode=False),
+                    "image_action": DatasetImage(decode=False),
+                    "image_candidate_states": List(DatasetImage(decode=False)),
+                }
+            else:
+                columns = {"image_actions": List(DatasetImage(decode=False))}
+            for name, feature in columns.items():
+                data = data.cast_column(name, feature)
+        self.examples = []
+        for row in data.select(range(limit)):
+            row["metbench_domain"] = self.domain
+            row["target"] = row["correct_choice" if self.domain == "minecraft" else "final_state"]
+            row["source_row"] = int(row["example_id"].rsplit("-", 1)[1])
+            self.examples.append(row)
+        self.by_index = {str(row["source_row"]): row for row in self.examples}
+        self.data = pd.DataFrame(
+            {
+                "index": row["source_row"],
+                "question": f"MET-Bench {self.domain} ({self.modality})",
+                "answer": str(row["target"]),
+                "example_id": row["example_id"],
+            }
+            for row in self.examples
+        )
+
+    def build_prompt(self, line):
+        """Keep every image in its original position in the user message."""
+        if isinstance(line, int):
+            line = self.data.iloc[line]
+        row = self.by_index[str(line["index"])]
+        content = []
+        for part in build_messages(row, self.modality)[0]["content"]:
+            if part["type"] == "text":
+                content.append({"type": "text", "value": part["text"]})
+            else:
+                buffer = io.BytesIO()
+                part["url"].save(buffer, format="PNG")
+                data = buffer.getvalue()
+                path = Path(self.img_root) / (hashlib.sha256(data).hexdigest() + ".png")
+                path.parent.mkdir(parents=True, exist_ok=True)
+                if not path.exists():
+                    path.write_bytes(data)
+                content.append({"type": "image", "value": str(path.resolve())})
+        return content
+
+    def dump_image(self, line):
+        """Return the ordered image paths for framework inspection tools."""
+        return [p["value"] for p in self.build_prompt(line) if p["type"] == "image"]
+
+    def evaluate(self, eval_file, **judge_kwargs):
+        """Score completed model predictions with MET-Bench's domain scorer."""
+        predictions = load(eval_file)
+        if predictions["index"].duplicated().any():
+            raise ValueError("Prediction file contains duplicate example indices")
+        if set(predictions["index"].astype(str)) != set(self.by_index):
+            raise ValueError("Prediction file does not match the selected examples")
+        if (
+            predictions["prediction"]
+            .astype(str)
+            .str.contains(INFER_FAIL_MSG, regex=False)
+            .any()
+        ):
+            raise ValueError("Retry failed inference requests before scoring MET-Bench")
+        scores = []
+        for _, prediction in predictions.iterrows():
+            response = prediction["prediction"]
+            if pd.isna(response):
+                response = ""
+            scores.append(
+                score(self.by_index[str(prediction["index"])], str(response))
+            )
+        accuracy = sum(scores) / len(scores)
+        if self.domain == "chess":
+            lower, upper = clustered_ratio_interval([(value, 1) for value in scores])
+        else:
+            trials = len(scores)
+            z = 1.959963984540054
+            denominator = 1 + z * z / trials
+            center = (accuracy + z * z / (2 * trials)) / denominator
+            margin = z * math.sqrt(accuracy * (1 - accuracy) / trials + z * z / (4 * trials * trials)) / denominator
+            lower, upper = max(0.0, center - margin), min(1.0, center + margin)
+        result = pd.DataFrame(
+            [
+                {
+                    "domain": self.domain,
+                    "modality": self.modality,
+                    "Overall": 100 * accuracy,
+                    "ci_lower": 100 * lower if lower is not None else None,
+                    "ci_upper": 100 * upper if upper is not None else None,
+                    "examples": len(predictions),
+                }
+            ]
+        )
+        dump(result, str(Path(eval_file).with_suffix("")) + "_acc.csv")
+        return result
+
+
+class METBenchText(METBenchImage):
+    """The text modality of the same MET-Bench examples."""
+
+    MODALITY = "TEXT"
+    modality = "text"
+
+    def __init__(self, dataset="METBench_minecraft_text", limit=500):
+        super().__init__(dataset, limit=limit)