8000 feat: Add unified x86 / aarch64 (ARM) build for TRTLLM image by rmccorm4 · Pull Request #803 · ai-dynamo/dynamo · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

feat: Add unified x86 / aarch64 (ARM) build for TRTLLM image #803

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 7 commits into from
Apr 24, 2025
Merged

feat: Add unified x86 / aarch64 (ARM) build for TRTLLM image #803

merged 7 commits into from
Apr 24, 2025

Conversation

rmccorm4
Copy link
Contributor
@rmccorm4 rmccorm4 commented Apr 24, 2025 •
edited
Loading

Overview:

Adds ARM64 support to TRTLLM docker build via docker build ARGS supplied from --platform linux/arm64:

./container/build.sh --framework tensorrtllm --platform linux/arm64

By default, the build args are set to x86/amd64 equivalents.

Details

ARM

Manually verified build, run, and inference on an A100 Linux ARM machine

Build

BASE_IMAGE=gitlab-master.nvidia.com:5005/dl/ai-dynamo/dynamo-ci/rmccormick
BASE_IMAGE_TAG=trtllm_48db263_aarch64

docker pull ${BASE_IMAGE}:${BASE_IMAGE_TAG}

./container/build.sh \
  --framework tensorrtllm \
  --base-image ${BASE_IMAGE} \
  --base-image-tag ${BASE_IMAGE_TAG} \
  --platform linux/arm64

# NOTE: --base-image and --base-image-tag can be replaced with the
# resulting image build from running `./container/build_trtllm_image.sh`

Run

./container/run.sh -it --framework tensorrtllm

# Pre-reqs
nats-server -js &
etcd --listen-client-urls http://0.0.0.0:2379 --advertise-client-urls http://0.0.0.0:2379 &

# Serve
cd /workspace/examples/tensorrt_llm
dynamo serve graphs.agg:Frontend -f ./configs/agg.yaml &

Inference

curl localhost:8000/v1/chat/completions   -H "Content-Type: application/json"   -d '{
    "model": "deepseek-ai/DeepSeek-R1-Distill-Llama-8B",
    "messages": [
        {
            "role": "user",
            "content": "In the heart of Eldoria, an ancient land of boundless magic and mysterious creatures, lies the long-forgotten city of Aeloria. Once a beacon of knowledge and power, Aeloria was buried beneath the shifting sands of time, lost to the world for centuries. You are an intrepid explorer, known for your unparalleled curiosity and courage, who has stumbled upon an ancient map hinting at ests that Aeloria holds a secret so profound that it has the potential to reshape the very fabric of reality. Your journey will take you through treacherous deserts, enchanted forests, and across perilous mountain ranges. Your Task: Character Background: Develop a detailed background for your character. Describe their motivations for seeking out Aeloria, their skills and weaknesses, and any personal connections to the ancient city or its legends. Are they driven by a quest for knowledge, a search for lost familt clue is hidden."
        }
    ],
    "stream":false,

# {"id":"chatcmpl-fb9de2caa74046c683372e45d7fcb12f","choices":[{"index":0,"message":{"content":"Okay, so I need to develop a detailed background for my character in this Eldoria setting. The user provided a sample character, Kaelira,","refusal":null,"tool_calls":null,"role":"assistant","function_call":null,"audio":null},"finish_reason":"length","logprobs":null}],"created":1745459577,"model":"deepseek-ai/DeepSeek-R1-Distill-Llama-8B","service_tier":null,"system_fingerprint":null,"object":"chat.completion","usage":null}

Reproducibility

docker tag dynamo:latest-tensorrtllm gitlab-master.nvidia.com:5005/dl/ai-dynamo/dynamo-ci/rmccormick:dynamo_490282b3_trtllm_48db263_aarch64

docker push gitlab-master.nvidia.com:5005/dl/ai-dynamo/dynamo-ci/rmccormick:dynamo_490282b3_trtllm_48db263_aarch64

x86

Build

BASE_IMAGE=urm.nvidia.com/sw-tensorrt-docker/tensorrt-llm-staging/release
BASE_IMAGE_TAG=main

docker pull ${BASE_IMAGE}:${BASE_IMAGE_TAG}

./container/build.sh \
  --framework tensorrtllm \
  --base-image ${BASE_IMAGE} \
  --base-image-tag ${BASE_IMAGE_TAG}

# NOTE: --base-image and --base-image-tag can be replaced with the
# resulting image build from running `./container/build_trtllm_image.sh`

Run

# Same as ARM steps above

Inference

# Same as ARM steps above

Reproducibility

docker tag dynamo:latest-tensorrtllm gitlab-master.nvidia.com:5005/dl/ai-dynamo/dynamo-ci/rmccormick:dynamo_490282b3_trtllm_04_24_25_x86_64

docker push gitlab-master.nvidia.com:5005/dl/ai-dynamo/dynamo-ci/rmccormick:dynamo_490282b3_trtllm_04_24_25_x86_64

@copy-pr-bot copy-pr-bot
Copy link
copy-pr-bot bot commented Apr 24, 2025

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@rmccorm4
Copy link
Contributor Author
rmccorm4 commented Apr 24, 2025 •
edited
Loading

Manually verified build/run/test on both Linux ARM64 and x86 machines with this change, see PR description for details @saturley-hall @dmitry-tokarev-nv

@rmccorm4 rmccorm4 enabled auto-merge (squash) April 24, 2025 03:06
@rmccorm4 rmccorm4 disabled auto-merge April 24, 2025 03:06
nnshah1
nnshah1 reviewed Apr 24, 2025
View reviewed changes
nnshah1
nnshah1 reviewed Apr 24, 2025
View reviewed changes
nnshah1
nnshah1 approved these changes Apr 24, 2025
View reviewed changes
Copy link
Contributor
@nnshah1 nnshah1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - few suggestions / questions

@rmccorm4 rmccorm4 changed the title feat: Add unified x86/aarch64 build for TRTLLM image feat: Add unified x86 / aarch64 (ARM) build for TRTLLM image Apr 24, 2025
nv-anants
nv-anants reviewed Apr 24, 2025
View reviewed changes
Copy link
Contributor
@nv-anants nv-anants left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Question - Is there official or something x86 equivalent trtllm base image for arm build?

Also jfyi, triggered gitlab with trtllm job - should validate default x86 flow

rmccorm4
rmccorm4 commented Apr 24, 2025
View reviewed changes
@rmccorm4 rmccorm4 mentioned this pull request Apr 24, 2025
Open
8000
@rmccorm4 rmccorm4 merged commit c522253 into main Apr 24, 2025
6 checks passed
@rmccorm4 rmccorm4 deleted the rmccormick/arm64/trtllm branch April 24, 2025 17:29
@piotrm-nvidia piotrm-nvidia mentioned this pull request Apr 24, 2025
Closed
@rmccorm4 rmccorm4 mentioned this pull request Apr 26, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@nnshah1 nnshah1 nnshah1 approved these changes

@nv-anants nv-anants nv-anants approved these changes

@tanmayv25 tanmayv25 Awaiting requested review from tanmayv25 tanmayv25 is a code owner

@ptarasiewiczNV ptarasiewiczNV Awaiting requested review from ptarasiewiczNV ptarasiewiczNV is a code owner

@ishandhanani ishandhanani Awaiting requested review from ishandhanani ishandhanani is a code owner

@alec-flowers alec-flowers Awaiting requested review from alec-flowers alec-flowers is a code owner

@piotrm-nvidia piotrm-nvidia Awaiting requested review from piotrm-nvidia

@dmitry-tokarev-nv dmitry-tokarev-nv Awaiting requested review from dmitry-tokarev-nv

Assignees
No one assigned
Labels
size/M
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@rmccorm4 @nnshah1 @nv-anants
0