Early stop saving best checkpoints

[minwang-ai]

Hi all, do you know how to save the best model? Since pytorchlighting 's earlystop callback will monitor val_loss and if val_loss stop decreasing, it will stop training automaticlly. In this case, the checkpoint of the final model would be the final epoch (the val_loss starts to increase). Can I save epoch 5 or 6 (before val_loss increasing) as the best model?

[rohitgr7]

you can specify

ModelCheckpoint(monitor='val_loss', save_top_k=1, ...)

this will save the best model.

[rohitgr7]

yes

[minwang-ai]

After using checkpoints callback, I found that the checkpoints of all my experiments are saved in the dirpath. Before that, the checkpoints of each run are saved in the subfolder of tensorboard log folder, i.e., logs/exp_name/exp-version/checkpoints. How can I save checkpoints with exp_name when I use callback?

In the docs, it shows:  By default, dirpath is None and will be set at runtime to the location specified by Trainer’s default_root_dir or weights_save_path arguments, and if the Trainer uses a logger, the path will also contain logger name and version.

However, it does not contain logger name and version when I use checkpoint callback. Different experiments’s checkpoints are saved at same path.

[rohitgr7]

can you try setting ModelCheckpoint(dirpath=None, ...)?

[minwang-ai]

can you try setting ModelCheckpoint(dirpath=None, ...)?

Thank you ! The problem solved! (It would be better to update docs😊)

[rohitgr7]

I think it's already updated:
https://github.com/PyTorchLightning/pytorch-lightning/blob/09cf167237e867f1ec67a5db87e5a02c2cea4b69/pytorch_lightning/callbacks/model_checkpoint.py#L65-L69