Skip to content
This repository has been archived by the owner on Nov 3, 2023. It is now read-only.

Keep multiple checkpoints during training #4970

Closed
hamjam opened this issue Feb 24, 2023 · 4 comments · May be fixed by #4978
Closed

Keep multiple checkpoints during training #4970

hamjam opened this issue Feb 24, 2023 · 4 comments · May be fixed by #4978
Labels
stale

Comments

@hamjam
Copy link

hamjam commented Feb 24, 2023

Hi,
Is there an option or a way to keep for example 5 best checkpoints in the training process by using train_model.py and not just the best model checkpoint?
As far as I understood, there isn't any option to keep multiple checkpoints in train_model.py, and by any option group, the new checkpoint will be overwritten on the only last checkpoint that has been saved.
Should I add this feature and if it's required create a pull request or is there a logic behind the way train_model.py keeps the checkpoints?
@ahkarami
Copy link

@hamjam
I think the --save-after-valid option does what you need it to do.

@hamjam
Copy link
Author

hamjam commented Feb 27, 2023

@ahkarami
As in this part of train_model.py I don't think by setting this option, a new checkpoint file will be created. It would create a new checkpoint after the validation but the checkpoint will not be saved by a new name and it will be just overwritten on the previously saved checkpoint by the same name.

@klshuster
Copy link
Contributor

You are correct, the current logic overwrites the checkpoints for each validation. We would appreciate a pull request that achieves your request if you are able to submit one!

@hamjam hamjam mentioned this issue Mar 10, 2023
Open
@github-actions
Copy link

github-actions bot commented Apr 6, 2023

This issue has not had activity in 30 days. Please feel free to reopen if you have more issues. You may apply the "never-stale" tag to prevent this from happening.

@github-actions github-actions bot added the stale label Apr 6, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
stale
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Now multiple checkpoints will be saved after using -stopk option in train_model.py hamjam/ParlAI
3 participants
@klshuster @hamjam @ahkarami