feat(serve.py): add api_path parameter to cli options to allow custom API endpoint configuration #2080
Conversation
ba60cc1 to
883dda3
Compare
…ses to allow custom API endpoint configuration refactor(serve.py): improve code readability by formatting long lines and restructuring some method calls for clarity style(serve.py): format code for improved readability by removing unnecessary line breaks and aligning parameters fix(serve.py): resolve api_path to default value when not using OpenAI spec or stream to ensure server functionality fix(serve.py): set default api_path to "/predict" for run_server function to ensure consistent behavior when no path is provided refactor(serve.py): simplify api_path handling by removing unnecessary resolved_api_path variable
|
Hi @botirk38, I assume to integrate via the OpenAI SDK you would use the chat completions API, right? |
Take this a grain of salt, im nmew to LitGPT. So the open ai sdk for a completions endpoint looks for /chat/completions, right now the path is set to/predict because of the lit serve dependency, therefore I want to override the path to /chat/completions. |
|
Hi @botirk38, just a quick note — LitGPT already supports the OpenAI-compatible Example usage: litgpt serve HuggingFaceTB/SmolLM2-135M-Instruct --openai_spec trueThen you can hit it using the OpenAI SDK or curl: curl -X POST http://127.0.0.1:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "SmolLM2-135M-Instruct",
"messages": [{"role": "user", "content": "Hello! How are you?"}]
}'Will also update the docs soon to make this more visible. Hope this helps! |
Hi thats amazing here, thanks for the support. Do you think its still worth adding the api path override? Otherwise Ill close it |
|
Yes @botirk38, we can keep it — just need to update the PR description. |
| @@ -1,4 +1,3 @@ | |||
| # Copyright Lightning AI. Licensed under the Apache License 2.0, see LICENSE file. | |||
Currently users of the library cannot override the api path, this makes it hard to integrate with the openai sdk for example. This PR introduces a fix for this use case.