Use SimpleDistributedPerLayerClipping optimizer in hooks mode #750
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…rch#720) Summary: Pull Request resolved: pytorch#720 register_backward_hook is deprecated and may lead to errors in gradient calculation. We switch to the supported register_full_backward_hook. Differential Revision: D68562558 Reviewed By: HuanyuZhang
|
This pull request was exported from Phabricator. Differential Revision: D72420168 |
iden-kalemaj
added a commit
to iden-kalemaj/opacus
that referenced
this pull request
Apr 3, 2025
…h#750) Summary: We use SimpleDistributedPerLayerOptimizer instead of DistributedPerLayerOptimizer. The latter causes an issue when switching to `register_full_backward_hook`. The issue arises because DistributedPerLayerOptimizer uses per-parameter hooks on top of the per-module hooks. During the backward pass, the per-parameter hooks fire before the per-module hooks. Per-sample gradients are computed when the per-module hooks fire, and an error occurs when the per-parameter hooks try to access the per-sample gradients before they are computed. Forcing the order in which hooks are called is not possible with PyTorch. Differential Revision: D72420168
3f2cbff to
72071ad
Compare
…h#750) Summary: Pull Request resolved: pytorch#750 We use SimpleDistributedPerLayerOptimizer instead of DistributedPerLayerOptimizer. The latter causes an issue when switching to `register_full_backward_hook`. The issue arises because DistributedPerLayerOptimizer uses per-parameter hooks on top of the per-module hooks. During the backward pass, the per-parameter hooks fire before the per-module hooks. Per-sample gradients are computed when the per-module hooks fire, and an error occurs when the per-parameter hooks try to access the per-sample gradients before they are computed. Forcing the order in which hooks are called is not possible with PyTorch. Differential Revision: D72420168
|
This pull request was exported from Phabricator. Differential Revision: D72420168 |
72071ad to
658b9ed
Compare
|
This pull request has been merged in 58f11ec. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
We use SimpleDistributedPerLayerOptimizer instead of DistributedPerLayerOptimizer.
The latter causes an issue when switching to
register_full_backward_hook.The issue arises because DistributedPerLayerOptimizer uses per-parameter hooks on top of the per-module hooks. During the backward pass, the per-parameter hooks fire before the per-module hooks. Per-sample gradients are computed when the per-module hooks fire, and an error occurs when the per-parameter hooks try to access the per-sample gradients before they are computed. Forcing the order in which hooks are called is not possible with PyTorch.
Differential Revision: D72420168