Hi, thank you for releasing the code and checkpoints. I have a follow-up question about how the released checkpoints map to the training stages described in the paper.
My understanding is that the paper's Base model is pretrained without reward labels, with the reward prediction and behavior cloning heads introduced afterward. When inspecting the released base/dynamics.pt, I noticed that its metadata includes reward_weight=1.0 and bc_weight=1.0, and that the checkpoint contains both heads. I may be misunderstanding how the release name maps to the paper's terminology.
Could you please clarify:
- Which training stage does the released
base checkpoint correspond to, and how do base, coverage_aware, and combined map to the stages reported in the paper?
- Were the reward and BC heads in the released
base checkpoint already trained? If so, any brief information about that training stage would be very helpful.
- Is the reward-free Base checkpoint described in the paper available?
This would help ensure that the reported stages are reproduced and compared correctly. Thank you for your time!
Hi, thank you for releasing the code and checkpoints. I have a follow-up question about how the released checkpoints map to the training stages described in the paper.
My understanding is that the paper's Base model is pretrained without reward labels, with the reward prediction and behavior cloning heads introduced afterward. When inspecting the released
base/dynamics.pt, I noticed that its metadata includesreward_weight=1.0andbc_weight=1.0, and that the checkpoint contains both heads. I may be misunderstanding how the release name maps to the paper's terminology.Could you please clarify:
basecheckpoint correspond to, and how dobase,coverage_aware, andcombinedmap to the stages reported in the paper?basecheckpoint already trained? If so, any brief information about that training stage would be very helpful.This would help ensure that the reported stages are reproduced and compared correctly. Thank you for your time!