Query Engine with Pydantic Outputs for vLLM/OpenAILike

Question

Hey, I'm running into an issue using structured outputs with query engine (following guide called Query Engine with Pydantic Outputs). I'm using vLLM for my backend and OpenAILike module for connecting. Currently getting Error 400, Value Error 'tool_choice' must either be a named tool, "auto", or "none". Saw a recommendation to add tool_choice="none" if using structured_llm.chat(), but since I'm using query engine I can't do that. How do I best do this? Do I have to manually create a workflow for this?

Logan M · Answer

Hmm, it looks like to me vllm is not supporting tool choice properly. The code is setting tool_choice to the name your tool (i.e. the output class)

Logan M · Answer

If vllm doesn't want to support that, you'll need to set is_function_calling_model=False

CaptainMario · Answer

That's odd since VLLM does support tool choice when I implemented the ReAct Workflow with the same config. I'll try this out. Thanks!

Logan M · Answer

React doesn't use structured output though 🤔 so tool choice wouldn't be used in that case

CaptainMario · Answer

I meant I used tools for a similar implementation of "Workflow for a ReAct Agent" that implements ReAct agent with Query Engine Tools. I'm assuming that uses tool choice for function calling in the same way that structured outputs use tool choice. I might be misunderstanding how this works. Thanks!

CaptainMario · Answer

Just got to test this today, seems like if is_function_calling_model=False the error changes to 'Value error, When using 'tool_choice', 'tools' must be set.'

Find answers from the community

Query Engine with Pydantic Outputs for vLLM/OpenAILike