Policy Collapse and Gibberish Rollouts in Search-R1 Early Training when TIS is Enabled (3B Model) #1533
Description
When reproducing Search-R1 using a 3B model on the Slime framework, I encountered a policy collapse during the early stages of training specifically when Token Importance Sampling (TIS) is enabled. I followed the tutorial settings in Enabling TIS (Trajectory Importance Sampling)
The model starts to generate chaotic, multilingual gibberish and completely ignores the system prompt and formatting constraints (e.g., and tags).
Comparison: TIS ON vs. OFF
TIS Enabled: The model collapses within the first few rollouts. Outputs become high-entropy "word salad" (as shown in the logs below).
TIS Disabled: Training is stable. The model follows the prompt instructions and converges normally.
In the early rollouts, the model fails to maintain the reasoning format and generate abnormal outputs:
