8000 Feature: Adds support for OpenAISpec in litgpt serve by bhimrazy · Pull Request #1943 · Lightning-AI/litgpt · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Feature: Adds support for OpenAISpec in litgpt serve #1943

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 53 commits into from
Apr 25, 2025
Merged

Feature: Adds support for OpenAISpec in litgpt serve #1943

merged 53 commits into from
Apr 25, 2025

Conversation

bhimrazy
Copy link
Contributor
@bhimrazy bhimrazy commented Feb 23, 2025 •
edited
Loading

What does this PR do?

feat: Add support for OpenAISpec in litgpt serve

Description:

  • Integrates OpenAISpec from LitServe to enable OpenAI-compatible API endpoints.
  • Allows seamless interaction with tools and applications expecting the OpenAI API format.
  • Reduces friction for users integrating LitGPT into their existing workflows or clients.

Fixes: #1908
Also closes: #1865

Usage

Step 1: Serve a model with --openai_spec enabled

litgpt download HuggingFaceTB/SmolLM2-135M-Instruct  
litgpt serve HuggingFaceTB/SmolLM2-135M-Instruct --openai_spec true

Step 2: Test with curl

curl -X POST http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "SmolLM2-135M-Instruct",
    "messages": [{"role": "user", "content": "Hello! How are you?"}]
  }'

@bhimrazy bhimrazy changed the title Add OpenAISpecLitAPI class and update run_server to support OpenAI specification Feature: Adds support for OpenAISpec Feb 23, 2025
@bhimrazy
Copy link
Contributor Author

image
Seems like HF_TOKEN has expired.

@bhimrazy bhimrazy changed the title Feature: Adds support for OpenAISpec Feature: Adds support for OpenAISpec in litgpt serve Feb 23, 2025
Borda
Borda reviewed Feb 24, 2025
View reviewed changes
Copy link
Member
@Borda Borda left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

overall looks good o me :)
looking at the failure due to OSError: There was a specific connection error when trying to load ...

@bhimrazy
Copy link
Contributor Author

Still getting the same error

Invalid credentials in Authorization header
FAILED tests/test_tokenizer.py::test_tokenizer_against_hf[SmolLM2-360M-Instruct] - OSError: There was a specific connection error when trying to load HuggingFaceTB/SmolLM2-360M-Instruct:
401 Client Error: Unauthorized for url: https://huggingface.co/HuggingFaceTB/SmolLM2-360M-Instruct/resolve/main/tokenizer.json (Request ID: Root=1-67beaf1b-4b40283979e3f7f365610da8;099fe274-d432-4d45-91f2-75f4ed03a73d)

Invalid credentials in Authorization header
FAILED tests/test_tokenizer.py::test_tokenizer_against_hf[SmolLM2-1.7B] - OSError: There was a specific connection error when trying to load HuggingFaceTB/SmolLM2-1.7B:
401 Client Error: Unauthorized for url: https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B/resolve/main/tokenizer.json (Request ID: Root=1-67beaf1c-44845ccf5d66f966550aef7a;e939c3a0-b641-4921-8396-4784713c51fc)

Invalid credentials in Authorization header
FAILED tests/test_tokenizer.py::test_tokenizer_against_hf[SmolLM2-1.7B-Instruct] - OSError: There was a specific connection error when trying to load HuggingFaceTB/SmolLM2-1.7B-Instruct:
401 Client Error: Unauthorized for url: https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B-Instruct/resolve/main/tokenizer.json (Request ID: Root=1-67beaf1f-474ff6b66337276b61106558;2463aeca-cfbb-49ab-80d0-fd87e331ec91)

fyi: @Borda @t-vi

@Borda
Copy link
Member
Borda commented Feb 27, 2025

Still getting the same error

looking into it and will debug it in #1940

@Borda Borda added the enhancement New feature or request label Mar 12, 2025
@craigpfeifer craigpfeifer mentioned this pull request Mar 14, 2025
Closed
@craigpfeifer
Copy link

Isn't the sticking point here that the OpenAI API was added in LitServe 0.2.5 , but litgpt currently uses 0.2.4 since 0.2.5 causes tests to fail?

@bhimrazy
Copy link
Contributor Author
bhimrazy commented Mar 14, 2025 •
edited
Loading

Isn't the sticking point here that the OpenAI API was added in LitServe 0.2.5 , but litgpt currently uses 0.2.4 since 0.2.5 causes tests to fail?

@craigpfeifer, we had the OpenAI Chat API in LitServe with the OpenAISpec well before that (in 0.1.1). I believe the OpenAI Embedding API was added in LitServe 0.2.5.

@craigpfeifer
Copy link

@craigpfeifer, we had the OpenAI Chat API in LitServe with the OpenAISpec well before that (in 0.1.1). I believe the OpenAI Embedding API was added in LitServe 0.2.5.

You are correct, seems like the streaming was fixed in 0.2.5 , but is present in earlier versions : Lightning-AI/LitServe#360

Borda
Borda reviewed Mar 18, 2025
View reviewed changes
Copy link
Member
@Borda Borda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we pls add a test?
also shall we then bump litServe to be 0.2.5+?

@bhimrazy
Copy link
Contributor Author

can we pls add a test? also shall we then bump litServe to be 0.2.5+?

Sure

@bhimrazy bhimrazy marked this pull request as ready for review April 15, 2025 17:51
@bhimrazy
Copy link
Contributor Author

All tests are finally passing, and the PR is ready for your review.

Just a quick note — I had to pin the pydantic version, as it was defaulting to an older release that didn’t have support for .model_copy.

Thanks so much! 😊
cc: @Borda @t-vi

@bhimrazy bhimrazy requested a review from Borda April 15, 2025 19:35
aniketmaurya
aniketmaurya approved these changes Apr 22, 2025
View reviewed changes
t-vi
t-vi approved these changes Apr 25, 2025
View reviewed changes
Copy link
Collaborator
@t-vi t-vi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @bhimrazy @Borda @craigpfeifer @aniketmaurya

@t-vi t-vi merged commit 54fe4a8 into Lightning-AI:main Apr 25, 2025
15 checks passed
@bhimrazy bhimrazy deleted the feat/add-support-for-openai-spec branch April 25, 2025 06:28
@craigpfeifer
Copy link

@t-vi when will this get released?

@bhimrazy bhimrazy mentioned this pull request Jun 25, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@t-vi t-vi t-vi approved these changes

@aniketmaurya aniketmaurya aniketmaurya approved these changes

@lantiga lantiga Awaiting requested review from lantiga lantiga is a code owner

@Borda Borda Awaiting requested review from Borda Borda is a code owner

Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Add support for OpenAISpec in litgpt serve
5 participants
@bhimrazy @Borda @craigpfeifer @t-vi @aniketmaurya
0