An end-to-end guide for scaling and serving LLM application in production. This repo currently contains one such application: a retrieval-augmented generation (RAG) app for answering questions about supplied information.
We'll be using OpenAI to access ChatGPT models like gpt-3.5-turbo, gpt-4, etc. and Anyscale Endpoints to access OSS LLMs like Llama-2-70b. Be sure to create your accounts for both and have your credentials ready.
- Start a new Anyscale workspace on staging using an
g3.8xlargehead node (you can also add GPU worker nodes to run the workloads faster). - Use the
default_cluster_env_2.6.2_py39cluster environment. - Use the
us-east-1if you'd like to use the artifacts in our shared storage (source docs, vector DB dumps, etc.).
git clone https://github.com/ray-project/llm-applications.git . # git checkout -b goku origin/goku
git config --global user.name <GITHUB-USERNAME>
git config --global user.email <EMAIL-ADDRESS>Our data is already ready at /efs/shared_storage/goku/docs.ray.io/en/master/ (on Staging, us-east-1) but if you wanted to load it yourself, run this bash command (change /desired/output/directory, but make sure it's on the shared storage,
so that it's accessible to the workers)
git clone https://github.com/ray-project/llm-applications.git .Then set up the environment correctly by specifying the values in your .env file,
and installing the dependencies:
pip install --user -r requirements.txt
export PYTHONPATH=$PYTHONPATH:$PWD
pre-commit install
pre-commit autoupdatetouch .env
# Add environment variables to .env
OPENAI_API_BASE="https://api.openai.com/v1"
OPENAI_API_KEY="" # https://platform.openai.com/account/api-keys
ANYSCALE_API_BASE="https://api.endpoints.anyscale.com/v1"
ANYSCALE_API_KEY="" # https://app.endpoints.anyscale.com/credentials
DB_CONNECTION_STRING="dbname=postgres user=postgres host=localhost password=postgres"
source .env- Open rag.ipynb to interactively go through all the concepts and run experiments.
- Use the best configuration (in
serve.py) from the notebook experiments to serve the LLM.
python app/main.py- Query your service.
import json
import requests
data = {"query": "What is the default batch size for map_batches?"}
response = requests.post("http://127.0.0.1:8000/query", json=data)
print(response.text)- Shutdown the service
from ray import serve
serve.shutdown()