Discrepancy in Qwen3-VL-8B-Instruct evaluation on VLM4D / Request for evaluation code

[Zhu-Yakun]

Hi DSR_Suite Authors,

Thanks for the fantastic work on dynamic spatial reasoning and the comprehensive DSR_Suite framework!

I am currently working on reproducing some of the baseline results presented in your paper. Specifically, I am evaluating Qwen3-VL-8B-Instruct on the VLM4D benchmark using the Direct Output (DO) setting. However, I have noticed some discrepancies between my evaluation results and the numbers reported in your paper.

To help me align my experimental setup perfectly with yours, could you kindly clarify a few evaluation details?

• Input Frame Sampling: In Section 7.2 (Evaluation Details) of the supplementary material, it states: "During evaluation, we uniformly sample 32 frames from each input video as visual inputs for all VLMs." Could you confirm if this 32-frame uniform sampling strategy was strictly applied to the VLM4D benchmark evaluation as well?
• Prompting Details: Could you share the exact text prompt/template used for the Direct Output evaluation on VLM4D? (e.g., how the video frames and the question were formatted together).
• Evaluation Code: Do you have any plans to open-source the evaluation scripts for these benchmarks? Releasing the evaluation code would be incredibly helpful for the community to ensure fair comparisons, standardized benchmarking, and easier reproduction of your great work.

Thanks in advance for your time and clarification!

[StewieZhou]

Hi Zhu-Yakun,

Thanks for your interest in our work and let me respond to your question one by one:

• Input Frame Sampling: For VLM4D, we also uniformly sample 32 frames for videos. However, because all videos in our DSR-Train are 360*640 pixels, we also set the sampling frame size as 360*640, which may influence the performance compared to the official implementation of VLM4D.
• Prompting Details: In our evaluation, video tokens are in front of the text tokens. For prompts, the template is question+". Choose the closest answer from the following 4 candidates and only output your choice as one of A,B,C,D: "+"A: {answer_A}; B: {answer_B}; C:{answer_C}; D:{answer_D}".
• Evaluation Code: Thanks very much for your good suggestion. We did not realize that we should also open-source our evaluation code for other benchmarks before. We will try to open-source these scripts later.

Thanks for your comments.

[StewieZhou]

Hi Zhu-Yakun,

Sorry for the late update for the requested scripts because I'm busy with another project before. I have uploaded the scripts for evaluation on STI-Bench and VLM4D. You can refer to the new README and modify the paths to model and benchmark in these scripts to start evaluation.

Thanks for your comments again!