8000 feat: Add TensorRT-LLM example for dynamo serve/run by tanmayv25 · Pull Request #456 · ai-dynamo/dynamo · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

feat: Add TensorRT-LLM example for dynamo serve/run #456

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 14 commits into from
Apr 3, 2025
Merged

feat: Add TensorRT-LLM example for dynamo serve/run #456

merged 14 commits into from
Apr 3, 2025

Conversation

tanmayv25
Copy link
Contributor
@tanmayv25 tanmayv25 commented Apr 1, 2025 •
edited
Loading

Overview:

Adds back limited TensorRT-LLM examples that were removed by this commit: 9aedc85

These new examples follows dynamo serve sdk and can start the processes automatically for the user.

Details:

The TRT-LLM examples were removed because we wanted to put our best UX forward. The outdated examples are still residing in trtllm_llmapi_v1 branch. I am adding these examples back to the main repository so they are more accessible.

Currently, these changes are only working and tested on aggregated example on single-node single gpu deployments.

TODO:

  1. ±Support single-node multiple GPUs with MPI launches in dynamo serve±
  2. Support disaggregated serving using single node system.

Where should the reviewer start?

The components folder within the example. The example follows vLLM closely.

@copy-pr-bot copy-pr-bot
Copy link
copy-pr-bot bot commented Apr 1, 2025

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

nnshah1
nnshah1 reviewed Apr 1, 2025
View reviewed changes
nnshah1
nnshah1 reviewed Apr 1, 2025
View reviewed changes
nnshah1
nnshah1 reviewed Apr 1, 2025
View reviewed changes
nnshah1
nnshah1 reviewed Apr 1, 2025
View reviewed changes
ryanolson
ryanolson requested changes Apr 1, 2025
View reviewed changes
nnshah1
nnshah1 reviewed Apr 1, 2025
View reviewed changes
nnshah1
nnshah1 reviewed Apr 1, 2025
View reviewed changes
nnshah1
nnshah1 reviewed Apr 1, 2025
View reviewed changes
@tanmayv25 tanmayv25 requested review from nnshah1 and ryanolson April 2, 2025 19:06
@tanmayv25 tanmayv25 requested a review from NVShreyas April 2, 2025 22:00
nnshah1
nnshah1 reviewed Apr 2, 2025
View reviewed changes
nnshah1
nnshah1 reviewed Apr 2, 2025
View reviewed changes
nnshah1
nnshah1 reviewed Apr 2, 2025
View reviewed changes
@tanmayv25 tanmayv25 requested a review from nnshah1 April 2, 2025 22:13
nnshah1
nnshah1 approved these changes Apr 2, 2025
View reviewed changes
Copy link
Contributor
@nnshah1 nnshah1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

great start! will save more clean up for future todo

@tanmayv25 tanmayv25 enabled auto-merge (squash) April 2, 2025 23:47
NVShreyas
NVShreyas reviewed Apr 3, 2025
View reviewed changes
@tanmayv25 tanmayv25 merged commit fe597df into main Apr 3, 2025
6 checks passed
@tanmayv25 tanmayv25 deleted the tanmayv-trtllm branch April 3, 2025 19:08
kylehh pushed a commit to kylehh/dynamo that referenced this pull request Apr 11, 2025
@tanmayv25 @nnshah1
feat: Add TensorRT-LLM example for dynamo serve/run (ai-dynamo#456)
6eb1054 
Co-authored-by: Neelay Shah <neelays@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@NVShreyas NVShreyas NVShreyas left review comments

@ryanolson ryanolson ryanolson approved these changes

@nnshah1 nnshah1 nnshah1 approved these changes

@hutm hutm Awaiting requested review from hutm hutm is a code owner

@biswapanda biswapanda Awaiting requested review from biswapanda biswapanda is a code owner

@ishandhanani ishandhanani Awaiting requested review from ishandhanani ishandhanani is a code owner

@julienmancuso julienmancuso Awaiting requested review from julienmancuso julienmancuso is a code owner

@hhzhang16 hhzhang16 Awaiting requested review from hhzhang16 hhzhang16 is a code owner

@piotrm-nvidia piotrm-nvidia Awaiting requested review from piotrm-nvidia piotrm-nvidia is a code owner

@ptarasiewiczNV ptarasiewiczNV Awaiting requested review from ptarasiewiczNV ptarasiewiczNV is a code owner

@grahamking grahamking Awaiting requested review from grahamking grahamking is a code owner

@paulhendricks paulhendricks Awaiting requested review from paulhendricks paulhendricks is a code owner

@tmonty12 tmonty12 Awaiting requested review from tmonty12 tmonty12 is a code owner

@GuanLuo GuanLuo Awaiting requested review from GuanLuo GuanLuo is a code owner

@rmccorm4 rmccorm4 Awaiting requested review from rmccorm4 rmccorm4 is a code owner

@tedzhouhk tedzhouhk Awaiting requested review from tedzhouhk tedzhouhk is a code owner

@alec-flowers alec-flowers Awaiting requested review from alec-flowers alec-flowers is a code owner

@kkranen kkranen Awaiting requested review from kkranen kkranen is a code owner

@sshchoi sshchoi Awaiting requested review from sshchoi sshchoi is a code owner

@PeaBrane PeaBrane Awaiting requested review from PeaBrane PeaBrane is a code owner

@mohammedabdulwahhab mohammedabdulwahhab Awaiting requested review from mohammedabdulwahhab mohammedabdulwahhab is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@tanmayv25 @ryanolson @nnshah1 @NVShreyas
0