8000 Test weight initialization for FSDP2 · Issue #342 · Modalities/modalities · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Test weight initialization for FSDP2 #342

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
le1nux opened this issue Apr 8, 2025 · 0 comments · Fixed by #351
Closed

Test weight initialization for FSDP2 #342

le1nux opened this issue Apr 8, 2025 · 0 comments · Fixed by #351
Assignees
@le1nux

Comments

@le1nux
Copy link
Member
le1nux commented Apr 8, 2025

In FSDP1, the model instantiation has the following consecutive steps:

Model instantiation (CPU memory) -> weight initialization -> FSDP1 Wrapping / Sharding (GPU memory)

The order in FSDP2 is different (weight init and wrapping are swapped):

Model instantiation (meta device or CPU memory) -> FSDP2 Wrapping / Sharding (meta device or CPU memory) -> weight initialization (GPU memory)

A test should make sure that the weight initialization (which is the same implementation for FSDP1 and FSDP2) initializes the weights following the specified distributions.

modalities/src/modalities/models/model_factory.py

Line 188 in b620f43

def get_weight_initalized_model(model: nn.Module, model_initializer: ModelInitializationIF) -> nn.Module:

@le1nux le1nux self-assigned this Apr 13, 2025
@le1nux le1nux linked a pull request Apr 14, 2025 that will close this issue
FSDPX Weight Init Tests #351
Merged
6 tasks
@le1nux le1nux closed this as completed May 22, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@le1nux le1nux

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

FSDPX Weight Init Tests Modalities/modalities
1 participant
@le1nux
0