fix(sdk): Using correct entrypoint for mpirun #2552
Conversation
Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>
Pull Request Test Coverage Report for Build 13990912640Warning: This coverage report may be inaccurate.This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.
Details
💛 - Coveralls |
Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>
|
cc @vsoch in case you are interested how do we place user's code into distributed MPI nodes using |
sdk/kubeflow/trainer/utils/utils.py
Outdated
| container_command = runtime.trainer.entrypoint | ||
| python_entrypoint = "python" | ||
| # mpirun uses file from this location: /home/mpiuser/<FILE_NAME>.py | ||
| func_file = os.path.join("/home", "mpiuser", func_file) |
There was a problem hiding this comment.
In case of user container image, this mpiuser directory does not exist.
So, could you open an issue to address arbitrary USER instead of this fixed mpiuser?
There was a problem hiding this comment.
It might be tricky for us to support arbitrary USER name here, since SDK user might don't know the structure of Docker container.
For now, could we say that the MPI-based runtimes must define mpiuser to run it ?
I am planning to use this user in the DeepSpeed runtime.
There was a problem hiding this comment.
For now, yes. We can enforce the USER, mpiuser.
However, in the future, we want to support switching base images since upstream can not support all of NVIDIA CUDA and the base images.
There was a problem hiding this comment.
Good point, I added the env variable that user can overrides in case directory is different: 39e50f1
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: tenzen-y The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>
|
/hold cancel |
* fix(sdk): Using correct entrypoint for mpirun Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Fix torchrun entrypoint Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Allow to configure mpiuser home dir Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> --------- Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>
* fix(sdk): Using correct entrypoint for mpirun Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Fix torchrun entrypoint Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Allow to configure mpiuser home dir Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> --------- Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>
* fix(sdk): Using correct entrypoint for mpirun Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Fix torchrun entrypoint Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Allow to configure mpiuser home dir Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> --------- Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>
* fix(sdk): Using correct entrypoint for mpirun Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Fix torchrun entrypoint Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> * Allow to configure mpiuser home dir Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> --------- Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>
For the MPI use-cases the Python file must be available on all Pods: Launcher + Node.
Thus, we should modify the command and args compare to
torchrun.After this change, the container command and args looks as follows for
mpirun:/assign @tenzen-y @kubeflow/wg-training-leads @Electronic-Waste @astefanutti @saileshd1402