Very Large Language Model Structured Outputs

Question

vLLM Structured Outputs.

Hi, I'm trying to do the same thing as this person (Issue #17677 on GitHub) but running into errors. If I do sllm.complete(prompt) or sllm.chat(ChatMessage[]), I get 'tool_choice must either be a named tool, "auto" or "none".'.

If I put tool_choice = auto or none, I get "Expected at least one tool call, but got 0 tool calls."

I copied the code in the documentation as well as the one recommended on GitHub issues. What could be the problem?

Also tried it with is_function_calling_model=True and False.

CaptainMario · Answer

Here is the similar issue I am referring to: https://github.com/run-llama/llama_index/issues/17677

Logan M · Answer

Could be a bug? Vllm updated something? Are you using openailike? (If you are, seems like vllm isn't following openai spec for some reason?)

Logan M · Answer

It's going to need a deeper debug -- looking at the actual api call being made and comparing that to whatever vllm expects

Logan M · Answer

Im out this week though, so I can't look into it atm

CaptainMario · Answer

Ok thanks. Yeah seems like when I do the .complete() it doesn't send the Pydantic struct, when I do .chat() it does format the input correctly but the response isn't a tool call. I think it may be a problem with the actual API call.

CaptainMario · Answer

Figured it out. Seems like vLLM is doing structured outputs without using tool calling. They're using client.beta.chat.completions.parse()

Logan M · Answer

Ah I see. I really need to update the openai llm class to use that

frandlied · Answer

I ran into the same issue. How did you solve it? I'm trying to get structured output from a model running in vLLM and get the error above.

CaptainMario · Answer

I created a class that inherited from OpenAILike. Then used self._get_client() to get the openaiclient. Then you can do client.beta.chat.completions.parse(...)

Find answers from the community

Very Large Language Model Structured Outputs