8000 feat: allow users to add env vars to dynamo deployment by hhzhang16 · Pull Request #862 · ai-dynamo/dynamo · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

feat: allow users to add env vars to dynamo deployment #862

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 5 commits into from
Apr 30, 2025
Merged

feat: allow users to add env vars to dynamo deployment #862

merged 5 commits into from
Apr 30, 2025

Conversation

hhzhang16
Copy link
Contributor
@hhzhang16 hhzhang16 commented Apr 28, 2025 •
edited
Loading

Overview:

Allow users to add environment variables to their Dynamo deployments

Details:

Allow support for env variables (using --env)

Where should the reviewer start?

Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)

@copy-pr-bot copy-pr-bot
Copy link
copy-pr-bot bot commented Apr 28, 2025

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

julienmancuso
julienmancuso approved these changes Apr 28, 2025
View reviewed changes
Copy link
Contributor
@julienmancuso julienmancuso left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

mohammedabdulwahhab
mohammedabdulwahhab reviewed Apr 28, 2025
View reviewed changes
Copy link
Contributor
@mohammedabdulwahhab mohammedabdulwahhab left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://github.com/ai-dynamo/dynamo/blob/main/examples/llm/configs/disagg.yaml#L35

^ If we added an envs section at the line ive linked above, it will set an env variable for that service at runtime. Futhermore, it's scoped specifically to that service. We can also use a cli arg to do the same. Can you test the following first?

dynamo serve ... --ServiceA.ServiceArgs.envs.HF_CACHE=$KEY
dynamo deploy ... --ServiceA.ServiceArgs.envs.HF_CACHE=$KEY

@hhzhang16
Copy link
Contributor Author

https://github.com/ai-dynamo/dynamo/blob/main/examples/llm/configs/disagg.yaml#L35

It doesn't seem to work 😬

(I'm working with agg.yaml for the record). I added an envs section to the config file then ran dynamo deployment create $DYNAMO_TAG -n $DEPLOYMENT_NAME -f ./configs/agg.yaml --VllmWorker.ServiceArgs.envs.HF_TOKEN=$HF_TOKEN

Usage: dynamo deployment create [OPTIONS] PIPELINE
Try 'dynamo deployment create -h' for help.
╭─ Error ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ No such option: --VllmWorker.ServiceArgs.envs.HF_TOKEN

mohammedabdulwahhab
mohammedabdulwahhab requested changes Apr 28, 2025
View reviewed changes
@hhzhang16 @mohammedabdulwahhab
Update deploy/dynamo/sdk/src/dynamo/sdk/cli/deployment.py
ad32db0 
Co-authored-by: mohammedabdulwahhab <furkhan324@berkeley.edu>
Signed-off-by: hhzhang16 <54051230+hhzhang16@users.noreply.github.com>
@hhzhang16 hhzhang16 merged commit 942a0fb into main Apr 30, 2025
11 checks passed
@hhzhang16 hhzhang16 deleted the hannahz/stop-rate-limit-hf branch April 30, 2025 17:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mohammedabdulwahhab mohammedabdulwahhab mohammedabdulwahhab approved these changes

@julienmancuso julienmancuso julienmancuso approved these changes

@hutm hutm Awaiting requested review from hutm hutm is a code owner

@biswapanda biswapanda Awaiting requested review from biswapanda biswapanda is a code owner

@ishandhanani ishandhanani Awaiting requested review from ishandhanani ishandhanani is a code owner

@nnshah1 nnshah1 Awaiting requested review from nnshah1 nnshah1 is a code owner

@tanmayv25 tanmayv25 Awaiting requested review from tanmayv25

@piotrm-nvidia piotrm-nvidia Awaiting requested review from piotrm-nvidia

@ptarasiewiczNV ptarasiewiczNV Awaiting requested review from ptarasiewiczNV

@ryanolson ryanolson Awaiting requested review from ryanolson

@grahamking grahamking Awaiting requested review from grahamking

@paulhendricks paulhendricks Awaiting requested review from paulhendricks

@tmonty12 tmonty12 Awaiting requested review from tmonty12

@GuanLuo GuanLuo Awaiting requested review from GuanLuo

@rmccorm4 rmccorm4 Awaiting requested review from rmccorm4

@tedzhouhk tedzhouhk Awaiting requested review from tedzhouhk

@alec-flowers alec-flowers Awaiting requested review from alec-flowers

@kkranen kkranen Awaiting requested review from kkranen

@sshchoi sshchoi Awaiting requested review from sshchoi

@PeaBrane PeaBrane Awaiting requested review from PeaBrane

Assignees
No one assigned
Labels
size/S
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@hhzhang16 @mohammedabdulwahhab @julienmancuso
0