8000 Ability to add your own instances to pools/dstack · Issue #1100 · dstackai/dstack · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Ability to add your own instances to pools/dstack #1100

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
15 of 18 tasks
TheBits opened this issue Apr 8, 2024 · 1 comment · Fixed by #1115, #1173, #1175, #1178 or #1183
Closed
15 of 18 tasks

Ability to add your own instances to pools/dstack #1100

TheBits opened this issue Apr 8, 2024 · 1 comment · Fixed by #1115, #1173, #1175, #1178 or #1183
Assignees
@TheBits

Comments

@TheBits
Copy link
Contributor
TheBits commented Apr 8, 2024 •
edited
Loading

After we finish pool support, we will be ready to add our own servers to the pool. This is necessary for a particular set of users that have their own bare metal servers.

To add a server, we need to use the dstack pool add command in the same way as we add cloud instances with additional arguments:

dstack pool add \
    --remote \
    --remote-host 192.168.88.131 \
    --remote-port 22 \
    --ssh-user=root \
    --ssh-identity ~/ssh_key

I will explain the new arguments:
--remote - Adding a running server
--remote-host 192.168.88.131 and --remote-port 22 - Connection detils
--ssh-user=root and --ssh-identity ~/ssh_key.pub - Credentials for ssh

To add the server, we need to go through the following steps.

  • connect via ssh to the server
  • install dstack-shim
  • add dstack-shim.service to the systemd

After install, dstack-shim should retrieve server resources and save them to the InstanceModel on the dstack server.
This information about server resources will be provided to the user via "dstack pool ps" command.

SSH semantics

It is necessary to change the name the command to dstack pool add-ssh to add a remote instance. The add remote instance command must have the same semantics as the ssh command.

Add support for the next cli arguments:

  • Support the destination format [user@]host or ssh://[user@]host[:port]
  • -l login_name specifies the user
  • -i key specifies the private key-containing file.
  • -p port the connection port

Improvements

  • After running the command, it should output a message that includes the instance name of the added remote host.
  • Support the argument for the --region
  • Support the argument for the --pool
  • Give the newly added remote instance a random name.
  • Make a draft documentation.
  • Verify the installation of nvidia-docker.
  • Use the dstack/base image in docker run to verify the output of the nvidia-smi command.

multinode support

  • Create two instances on the same network, check that they connect to each other.
  • I need to create a new issue with support for remote instances as a backend for multinode.

Improvements

  • The add_remote (process_instances.py) function blocks the server. For example, Ctrl-C does not work.
  • Add the -w flag. Enabling it should show the process of provisioning the remote instance to the idle state. As an example, it should work as dstack ps -w.
  • Log a warning that the remote instance failed to provisioning
  • Add multionde documentation
  • I can add the same instance two times
@TheBits TheBits self-assigned this Apr 8, 2024
@r4victor
Copy link
Collaborator

@TheBits for multi-node support, dstack also needs to know VMs internal ips.

@TheBits TheBits mentioned this issue Apr 11, 2024
Merged
@TheBits TheBits linked a pull request Apr 11, 2024 that will close this issue
Added the ability of adding own instances #1115
Merged
@TheBits TheBits reopened this Apr 24, 2024
@TheBits TheBits linked a pull request Apr 26, 2024 that will close this issue
Create public ssh key is it not exist in dstack pool add-ssh #1173
Merged
@TheBits TheBits reopened this Apr 26, 2024
@TheBits TheBits linked a pull request Apr 26, 2024 that will close this issue
Fixed is the environment file upload #1175
Merged
@TheBits TheBits mentioned this issue Apr 26, 2024
Merged
@TheBits TheBits reopened this Apr 26, 2024
@TheBits TheBits linked a pull request Apr 26, 2024 that will close this issue
Fix bugs in dstack pool add-ssh #1178
Merged
@TheBits TheBits reopened this Apr 29, 2024
@TheBits TheBits linked a pull request Apr 29, 2024 that will close this issue
Improve the retries in the get_host_info and get_shim_healthcheck #1183
Merged
This was linked to pull requests May 1, 2024
Fix DSTACK_NODE_RANK #1189
Merged
Improvements for adding SSH instances #1202
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@TheBits TheBits

Labels
None yet
Projects
None yet
Milestone
No milestone
2 participants
@TheBits @r4victor
0