AdaptMMBench: Benchmarking Adaptive Multimodal Reasoning for Mode Selection and Reasoning Process

🔥News

• [2026/04/13] We released the evaluation code for AdaptMMBench.
• [2026/02/03] We released arxiv paper and data samples of AdaptMMBench. Welcome to download and explore them!

Overview

While adaptive multimodal reasoning shows great promise, current studies lack an independent evaluation of the mode selection mechanism and often neglect fine-grained process analysis. To address this gap, we propose AdaptMMBench, which isolates the assessment of mode selection capability by dynamically identifying model capability boundaries and utilizing the Matthews Correlation Coefficient (MCC). Furthermore, it facilitates a multi-dimensional process evaluation encompassing quality(key step coverage, tool effectiveness) and efficiency(reasoning turn, tool invocation frequency, token consumption). Our evaluation reveals that while adaptive mode selection scales with model capacity, it notably decouples from final accuracy. Furthermore, tool effectiveness exhibits significant inconsistency across different model architectures.

AdaptMMBench

Benchmark

AdaptMMBench encompasses 1,420 samples spanning five domains: real-world, OCR, GUI, math, and knowledge. To ensure a comprehensive evaluation, the dataset spans a range of difficulty levels, balancing tasks that can be solved through text-only reasoning with those that require adaptive tool invocation, such as zooming and image transformations (e.g., rotation and contrast adjustment). Benchmark quality is maintained through a rigorous multi-stage verification pipeline, in which human annotators and GPT-5 collaboratively validate bounding box annotations and key reasoning steps.

Metrics

$$ \text{MCC} = \frac{TP \cdot TN - FP \cdot FN} {\sqrt{(TP+FP)(TP+FN)(TN+FP)(TN+FN)} + \epsilon} $$

where $\epsilon$ is a small constant for numerical stability. MCC ranges from $[-1,1]$, with $1$ indicating perfect agreement with the optimal mode selection, $0$ denoting the chance-level performance, and $-1$ indicating complete misalignment.

Adaptive Mode Selection Evaluation

Adaptive intelligence depends on a model’s ability to judge whether its available information is sufficient to solve a task; therefore, reasoning mode selection should be evaluated independently of answer correctness. Under this principle, tasks solvable via text-only reasoning are labeled Tool-Redundant, while those requiring additional visual information are labeled Tool-Required. Mode selection is evaluated using a confusion matrix, where correct and incorrect tool invocation decisions correspond to TP, FN, TN, and FP cases.

Matthews Correlation Coefficient (MCC). Because the proportion of tool-redundant and tool-required cases varies across models, leading to class imbalance, we adopt MCC as a robust evaluation metric.

Reasoning Process Evaluation

We evaluate the reasoning process from both quality and efficiency perspectives. Reasoning quality is measured by key step coverage, which assesses logical alignment with human-annotated solution steps, and tool execution effectiveness, which evaluates whether each tool invocation correctly serves the intended purpose of its corresponding reasoning step and is executed without errors. Reasoning efficiency is assessed using the number of reasoning steps, tool invocation frequency, and total token consumption.

Inference & Evaluation

Before running inference or evaluation, please update the model settings in ./config/models and replace the API configuration below with your own credentials.

Run Inference

Use the following command to generate inference results for the adaptive, text, and oracle reasoning modes:

python -m adaptmmb.cli \
  'modes=["adaptive","text","oracle"]' \
  'stages=["infer"]' \
  'model_names=["gpt-5"]' \
  'judge.base_url=https://api.deerapi.com/v1' \
  'judge.api_key=YOUR_API_KEY' \
  'judge.model=gpt-5.2' \
  'dataset.data_path=./path/to/AdaptMMBench' \
  'dataset.image_root=./path/to/AdaptMMBench' \
  'runtime.out_dir=outputs/my_run'

Adaptive Mode Selection Evaluation (MCC)

After inference is completed, run judging and metric computation for the adaptive, text, and oracle modes:

python -m adaptmmb.cli \
  'modes=["adaptive","text","oracle"]' \
  'stages=["judge","metrics"]' \
  'model_names=["gpt-5"]' \
  'judge.base_url=https://api.deerapi.com/v1' \
  'judge.api_key=YOUR_API_KEY' \
  'judge.model=gpt-5.2' \
  'dataset.data_path=./path/to/AdaptMMBench' \
  'dataset.image_root=./path/to/AdaptMMBench' \
  'runtime.out_dir=outputs/my_run'

Process Evaluation

Use the following command to evaluate the reasoning process for the adaptive mode:

python -m adaptmmb.cli \
  'modes=["adaptive"]' \
  'stages=["process_judge"]' \
  'model_names=["gpt-5"]' \
  'judge.base_url=https://api.deerapi.com/v1' \
  'judge.api_key=YOUR_API_KEY' \
  'judge.model=gpt-5.2' \
  'dataset.data_path=./path/to/AdaptMMBench' \
  'dataset.image_root=./path/to/AdaptMMBench' \
  'runtime.out_dir=outputs/my_run'

Citation

If you find our project helpful, please consider citing it using the following reference:

@article{zhang2026adaptmmbench,
      title={AdaptMMBench: Benchmarking Adaptive Multimodal Reasoning for Mode Selection and Reasoning Process},
      author={Zhang, Xintong and Zhang, Xiaowen and Wu, Jongrong and Gao, Zhi and Yan, Shilin and Diao, Zhenxin and Gao, Kunpeng and Chen, Xuanyan and Wu, Yuwei and Jia, Yunde and others},
      journal={arXiv preprint arXiv:2602.02676},
      year={2026}
}