-
|
Hi! I'm having trouble using the instructor and need some help.
First, I confirmed that it works well when creating structured_output using instructor with OpenAI's API as shown below! from openai import OpenAI
from pydantic import BaseModel
class IntentSchema(BaseModel):
intent: str
def structured_output(instruction:str) -> object:
llm = OpenAI(
api_key='__my_openai_api_key__'
)
client = instructor.from_openai(llm)
res = client.chat.completions.create(
model="gpt-4o",
response_model=IntentSchema,
messages=[{"role": "user", "content": instruction}],
)
return res
prompt = """
Please intent cooridnating one of below:
- coffee
- beer
- tea
user question: ice americano
"""
structured_output(prompt)IntentModel(intent='coffee')The problem starts here. When I use vllm locally to convert the 'Qwen2.5-14B-Instruct-AWQ' model to OpenAI format and serve it so that it can be used, and then apply the instructor, a Connection Error occurs. from openai import OpenAI
from pydantic import BaseModel
class IntentSchema(BaseModel):
intent: str
def structured_output(instruction:str) -> object:
llm = OpenAI(
base_url="___local_api_url_vllm___",
api_key="__api_key__"
)
instructor_client = instructor.from_openai(
client=llm
)
res = instructor_client.chat.completions.create(
model="Qwen/Qwen2.5-14B-Instruct-AWQ",
response_model=IntentSchema,
messages=[
{"role": "user", "content": instruction}
],
temperature=0.0
)
return res
prompt = """
Please intent cooridnating one of below:
- coffee
- beer
- tea
user question: ice americano
"""
structured_output(prompt)InstructorRetryException: Connection error.If you have had similar experience or know a solution, please help..! |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment
-
|
I solved it. In my case, I found that instructor is not applied when the speculative decoding option is activated when serving local llm to vllm. I will close this discussion! |
Beta Was this translation helpful? Give feedback.
I solved it. In my case, I found that instructor is not applied when the speculative decoding option is activated when serving local llm to vllm. I will close this discussion!