[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[WIP][Infer] Inference Distributed RPC Framework Optimization #5756

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open

[WIP][Infer] Inference Distributed RPC Framework Optimization #5756

wants to merge 5 commits into from
+289 −60

Conversation

LRY89757
Copy link
Contributor
  1. Optimize the data path: from List->CPU Tensor->List->rpc_param->GPU Tensor to List->rpc_param->GPU Tensor
  2. Wrap the async forward only once
  3. Only rank0 Worker runs the sampler and returns the return value
  4. Pass the rpc param to worker 0 instead of all workers, and worker 0 broadcast the param to all workers using NCCL.

The performance is not good enough, which needs to be further optimized

@LRY89757 LRY89757 requested a review from a team as a code owner May 27, 2024 07:16
@LRY89757 LRY89757 changed the title [Infer] Inference Distributed RPC Framework Optimization [WIP][Infer] Inference Distributed RPC Framework Optimization May 27, 2024
@LRY89757 LRY89757 added the tensor-parallel related to the tensor-parallel feature label May 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
colossal-inference tensor-parallel related to the tensor-parallel feature
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@LRY89757