Inquiry regarding VLM4D baseline evaluation details and reproduction scripts Description

[Zhu-Yakun]

Hello MLLM-4D authors,

Thank you for the fantastic work on this paper and for sharing your insights on spatial-temporal reasoning!

I am currently working on reproducing the baseline evaluations on the VLM4D benchmark to better understand the improvements introduced by your framework. [cite_start]However, I have noticed a slight discrepancy between my reproduction of the baseline and the results reported in the paper (Table 2 reports the baseline at 52.1%, and the proprietary/open-source table reports Qwen3-VL-8B-Instruct at 52.2%).

Using my current evaluation pipeline, I am achieving an accuracy of 53.41%, which is roughly 1.2 points higher than the reported baseline.

For transparency, here are the exact configurations I used for my baseline evaluation:

• Prompting: Direct-Output (DO mode)
• Resolution Settings: Original resolution, bounded by min_pixels=64x28x28 and max_pixels=256x28x28
• Decoding Strategy: Greedy decoding (do_sample=False)
• Frame Sampling: Fixed to 32 frames (nframes=32). For short videos, it automatically caps at the actual frame count, floored to the nearest multiple of 2.

Could you please clarify if there are any specific differences in your evaluation script for the baselines? Specifically, I would love to know:

1. What frame sampling strategy and resolution limits were applied to the baseline models during your evaluation?
2. Are you planning to release the evaluation scripts used for the baselines?

Aligning these evaluation details will be incredibly helpful for my research. Thank you again for your time and for the great contribution to the community!

[flow0314]

Hi Yakun, thanks for your interest in our work!
We implemented our evaluation for VLM4D following their official codebase.
I will clean up the evaluation scripts used for VLM4D tonight and plan to open-source them tomorrow.

[Zhu-Yakun]

Hi MLLM-4D Authors,

Thank you so much for the prompt reply and for your commitment to open-sourcing the evaluation scripts! It is greatly appreciated.

I actually dug a bit deeper into the official VLM4D evaluation codebase and found the exact reason for the performance gap I was seeing. It turns out the official VLM4D evaluation script defaults to temperature=1 (stochastic sampling).

Since you strictly followed their official setup, your baseline results naturally reflect this. In my local reproduction, I was using deterministic greedy decoding (do_sample=False, temperature=0), which completely explains why my baseline evaluation yielded a slightly higher accuracy of 53.41% compared to the reported 52.2%.

This actually brings up an interesting discussion point regarding the evaluation protocol. Since temperature=1 introduces variance, it might not fully capture the model's true performance ceiling (or most confident prediction) for deterministic QA tasks, and single-run results can fluctuate. Out of curiosity, did you happen to run the evaluation multiple times and average the results for the baselines, or was it a single pass following the VLM4D default?

It is great to have this cleared up! Thank you again for your transparency and the fantastic work on this project.