Issues in router query engine

At a glance

The community member is facing issues with the RouterQueryEngine, where it is selecting the wrong QueryEngine to execute despite providing the right explanation of the tool it wants to select. The issue seems to be related to the index number returned by the engine, which is incorrect. The community members discuss potential solutions, such as using the tool name instead of the index, and modifying the PydanticSingleSelector class to better handle the selection process. However, there is no explicitly marked answer in the comments.

ddhiraj

I am facing issues in RouterQueryEngine, I have around 6 tools that I am passing to it, and I get some werid behaviour in which it provides right explanation of the tool that it want to select but it returns wrong index number. Due to this the RouterQueryEngine ends up choosing wrong QueryEngine to execute.

This doesn't happen reqularly but its quite frequent though. Any tips to solve this would be greatly appreciated

10 comments

LLogan M

seems like the llm is having trouble counting in the selector 🤔 What LLM are you using?

ddhiraj

yes its having hard time choosing the right index, i am using GPT-4 0613 version

LLogan M

ooc, how are you setting up the router?

ddhiraj

Plain Text

   
query_engine = RouterQueryEngine.from_defaults(
        query_engine_tools=[
            sql_tool,  # index=0
            vector_tool,  # index=1
            summary_tool,  # index=2
            fetch_pdf_tool,  # index=3
            zone_tool,  # index=4
            fallback_tool,  # index=5
        ],
        selector=PydanticSingleSelector.from_defaults(
            verbose=True, prompt_template_str=SELECT_TOOL_PROMPT
        ),
        summarizer=tree_summarizer,  # used when multi-selector is used
        verbose=True,
    )

ddhiraj

@Logan M - can you see anything wrong in the setup?

LLogan M

seems fine. Tbh looking at the code though, I'm not sure why predicting an index was chosen, predicting a name should be good enough?

You can do this yourself as well, which maybe is better?

Plain Text

tools_by_name = {tool.metadata.name: tool for tool in tools}

# can pass in a user msg or chat history
resp = llm.chat_with_tools(
  tools, 
  user_msg=f"Given this latest message, pick at least one tool to use: {user_msg}", 
  error_on_no_tool_call=False
)

tool_calls = llm.get_tool_calls_from_response(resp)
for tool_call in tool_calls:
  print(tool_call.tool_name)
  print(tool_call.tool_kwargs)
  tool_result = tools_by_name[tool_call.tool_name](**tool_call.tool_kwargs)
  print(tool_result)

ddhiraj

will a PR to do this change in Pydantic Single Selector help?

LLogan M

this is actually what the single selector is doing lol

The above code would let you run the selection yourself, and better handle errors

ddhiraj

yeah right, but the Single Selection Class is pretty basic
class SingleSelection(BaseModel):
"""A single selection of a choice."""

index: int
reason: str

Should be changing it to add the name of the tool also, so that it can predict correctly

LLogan M

oh right yea, i did mention that

Add a reply

Find answers from the community

Issues in router query engine