8000 fp16 support · Issue #377 · pytorch/opacus · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

fp16 support #377

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
ffuuugor opened this issue Mar 11, 2022 · 4 comments
Open

fp16 support #377

ffuuugor opened this issue Mar 11, 2022 · 4 comments
Labels
enhancement New feature or request

Comments

@ffuuugor
Copy link
Contributor

(possibly with autocast and loss scaling, but in our experience with Alex this may result in training instabilities). On the other hand, it reduces memory usage and increases speed (roughly a factor 2 for both).
@ffuuugor ffuuugor added the enhancement New feature or request label Mar 11, 2022
@yuxiang-guo
Copy link

Hi, I wonder whether opacus support fp16. In my experiment, I use the fp16 but get the error:

File "/home/yxguo/anaconda3/envs/ditto/lib/python3.7/site-packages/apex/amp/handle.py", line 110, in scale_loss
if not optimizer._amp_stash.params_have_scaled_gradients:
AttributeError: 'DPOptimizer' object has no attribute '_amp_stash'

@ffuuugor
Copy link
Contributor Author

Hi
As of this moment opacus doesn't support fp16.

This issue is created to track progress of adding the support. As of today it's not something planned for the near future (but we always welcome external pull requests)

@alidadsetan
Copy link

Hi. Any roadmap for this yet? This seems to be the main bottleneck, now that you have the "ghost clipping" in place.

@iden-kalemaj
Copy link
Contributor
iden-kalemaj commented Jun 2, 2025 •
edited
Loading

Hi @alidadsetan we are enabling support for mixed and fully low precision training with Opacus within the next two weeks. Please see this in progress PR: #764

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@ffuuugor @alidadsetan @yuxiang-guo @iden-kalemaj
0