This repository contains the infrastructure and bootstrap code for the vLLM continuous integration pipeline using Buildkite.

Current CI Infrastructure Setup:

• AWS Buildkite Elastic CI Stack: Infrastructure code in infra/aws
• 8 TPU Nodes on GCP: Infrastructure code in infra/gcp_old
• GKE Cluster on GCP (currently not in use): Infrastructure code in infra/gcp

Bootstrap scripts are located in the scripts/ directory.

vLLM leverages Buildkite for CI workflow. Whenever a commit is pushed to the vLLM GitHub repository, a Buildkite webhook triggers an event that initiates a new build in the Buildkite pipeline with relevant details like Github branch and commit.

Build Process Overview:

• Bootstrap Step:

  • Executed via scripts/ci_aws_bootstrap.sh.
  • Utilizes a CI Jinja2 template (scripts/test-template-aws.j2) along with the list of tests from vLLM to render a Buildkite YAML configuration that defines all build/test steps and their configurations.
  • Uploads the rendered YAML to Buildkite to initiate the build.
  • Note: We are transitioning to a custom Buildkite pipeline generator to replace the Jinja2 template rendering soon.

• Job Queueing and Execution:

  • Each Buildkite step is associated with an agent queue.
  • After uploaded, steps are pushed into the queue, waiting to be picked up by a Buildkite agent.

We use the Buildkite Elastic CI Stack to set up our autoscaling Buildkite agent cluster on AWS.

Components of the stack for each Agent Queue:

• AWS CloudFormation Stack:

  • Contains an EC2 Auto Scaling Group and an AWS Lambda function.

• EC2 Auto Scaling Group:

  • Automatically scales number of EC2 instances based on the workload from the Buildkite queue.
  • Each EC2 instance comes with a Buildkite agent that executes jobs.

• AWS Lambda Function:

  • Periodically polls Buildkite to assess capacity needs for the queue and adjusts the size of the Auto Scaling Group accordingly.