Add Cosine Annealing LR scheduler as a decay method #3507
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tgaddair
approved these changes
Aug 8, 2023
dennis-rall
pushed a commit
to dennis-rall/ludwig
that referenced
this pull request
Aug 9, 2023
dennis-rall
pushed a commit
to dennis-rall/ludwig
that referenced
this pull request
Aug 9, 2023
justinxzhao
added a commit
that referenced
this pull request
Aug 25, 2023
This reverts commit b0bba7c.
justinxzhao
added a commit
that referenced
this pull request
Aug 25, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Implements Cosine Annealing: https://arxiv.org/pdf/1608.03983.pdf
CosineAnnealingWithRestarts is a scheduling technique that starts with a very large learning rate and then aggressively decreases it to a value near 0 before increasing the learning rate again. This happens either at fixed intervals, or are growing intervals, both of which can be configured.
To enable cosine learning rate scheduling with warm restarts, you can set:
There are 3 optional parameters that you can configure:
T_0: The number of steps before warm restarts. Defaults tosteps_per_checkpointif not specified.T_mult: Period between warm restarts. Defaults to 1. If set to 2 or greater, it multiplies the current interval by the multiplier between warm restarts. For e.g., ifT_0is 20 steps andT_multis 2, then the periods between restarts will be 20, 40, 80, 160, ....eta_min: The minimum learning rate the LR should be decayed to. Defaults to 0.This PR also refactors the way the learning rate schedulers are configured, using a new SequentialLR object.