Experimental Setup Question

[wumingqi]

Thank you for your contribution to LLM research. I encountered some problems when replicating your work. Could you provide more detailed experimental details? For example, the generation parameters and prompts you used. Also, in Figure 1, how were the results of the "Before RL" experiment obtained? Could you explain your evaluation method? Thank you!

[mokeevdmitrii]

Hello! Would like to add, that in Dr. GRPO the authors achieve a score of 69.0 on MATH-500 using Qwen2.5-Math-7B without any template and few-shot prompting - hypothesising that Qwen-Math models are pretrained using raw instruct-response pairs.

However, they do not explore using different generation params for this evaluation, so I suppose the ceiling of the Qwen2.5-Math-7B model may be even higher (and RLVR could exchange pass@k for pass@1)

The authors of this paper explore this prompt-related phenomena in appendix F (well, they miss a bit and do not achieve 69.0, as the prompt is somewhat different).

So, maybe the problem with Qwens is indeed in the correct evaluation of the base model capabilities, which is (partially) caused by their prompt sensitivity.

Thank you for your work!

[RulinShao]

Hi @mokeevdmitrii @wumingqi, thanks for your insightful comments! We've added more experiments with different prompts to study the effect of prompting in RLVR, and showed that our conclusions hold across prompts. Please find detailed results in our arXiv paper or our second blog for more discussion.

In short, we also show that prompting can impact the "Prev RL" performance significantly for Qwen models, sometimes in an interpretable way—please refer to our spurious prompt blog for more details. Both spurious rewards and spurious prompts show that people should be careful about the improvements they see from RL—there are many superficial factors that can contribute to significant improvement. Establishing strong baselines and verifying results with a diverse set of models are very important for drawing conclusions.

To answer @wumingqi's questions: The "Pre-RL" scores are obtained using the same evaluation setup as we use during RL with ground-truth labels. The implementation was borrowed from the official OpenRLHF (originates from official Qwen's repository), and the evaluation code is included in this repository (code/ttrl/verifier/qwen/qwen_eval.py| eval cmds). For Fig. 1, we used the official Qwen default chat template, as has been commonly used in many previous works such as TTRL and 1-shot RL. In addition, we supplemented results with other prompts, such as prompts used by simpleRL-zoo and Sober.

Thanks again for your insightful questions and discussions! Please let us know if you have further questions!