Optimizer and trainer states for OLMo-7B (Feb. 2024)

[rahuln]

❓ The question

I'd like to be able to do continued pretraining with the intermediate checkpoints of the original OLMo-7B model (from February 2024), including initializing from the intermediate optimizer and trainer states. I found an older version of the README that includes instructions on how to access these intermediate checkpoints as well as a list of paths pointing to those files, but they don't seem to work with the OLMo training code (i.e., files like model.pt / model.safetensors and optim.pt / optim.safetensors can't be found at those links). Is there somewhere I can go to find links to the original OLMo checkpoints with optimizer and trainer state files?

[aman-17]

Hey @rahuln, those files were deleted by mistake. I'm in the process of restoring them. I will update you once they are back.

[rahuln]

I just wanted to follow up on this, are those training checkpoints now available?

[aman-17]

Hey @rahuln, I have started the process, it might take 3-4 days to upload. If you need any particular checkpoint, I can fast-forward that and provide you the link.

[rahuln]

Awesome, thank you! I'd mainly like to use earlier checkpoints (the first 5 or so), so if those are available sooner and you're able to share the links I'd really appreciate it.

[aman-17]

Hey @rahuln, first 5 checkpoints are restored. Check out the same links.

[rahuln]

Great, thanks so much! Would you happen to have the links for all of the early checkpoints? The links in the CSV file for the original version of OLMo (this file) only start at step 5000.

[aman-17]

Hey @rahuln, yes we do have them, I will upload them by the end of this week.