Workflow for fine-tuning from a pretrained checkpoint

[alif-munim]

Hello, I have a dataset of 336x336 videos, and I had a few questions about how to use the pretrained checkpoints as a starting point (currently trying to use vitg 384).

I see an anneal_ckptoption in the cooldown configs, but when I set that to one of the vit checkpoints provided in the README, it does not actually seem to be loaded. Rather, the training just starts from scratch. I also don't see any code in the repository where the anneal_ckpt option is being used.

Any help with this would be greatly appreciated!

[ballasnicolas]

Hi!

Here is a config that continue the training of an existing VIT.g/16 checkpoint: https://github.com/facebookresearch/vjepa2/blob/main/configs/train/vitg16/cooldown-384px-64f.yaml. You will need to specify the location of the pretrained checkpoint here: https://github.com/facebookresearch/vjepa2/blob/main/configs/train/vitg16/cooldown-384px-64f.yaml#L87

[alif-munim]

Hi @ballasnicolas, thanks for your reply! My issue with this is that it seemed training was not actually starting from the specified pretrained checkpoint. Is there a reference somewhere in the code on where this anneal_ckpt variable is used?

[mmuckley]

Hello @alif-munim, it looks like the particular line for loading a checkpoint for annealing got removed during the code cleaning process for open-source release. In our internal code, we have some lines like this:

    if load_model:
        if is_anneal:
            if os.path.exists(latest_path) and resume_anneal:
                load_path = latest_path
            else:
                load_path = anneal_ckpt
                resume_anneal = False
        else:
            load_path = r_file if r_file is not None else latest_path

I can look a little bit into making a PR to get this into the open-source release, but I want to try to check that I don't break other code in the process, so it might take a bit.

[mmuckley]

Hello @alif-munim, if you get a chance would be willing to see if PR #72 works for you?