My original user input query is getting

JJoshhhh

My original user input query is getting transformed before it hits my SubQuestionQueryEngine and I’m stumped as to why. Does OpenAIAgent.from_tools transform the user’s input query_str?

For example:

User’s input question: List all of the insurance providers covering medical costs for the patient, including entities providing Letters of Protection (LOP)
Is being changed to: insurance providers
Which gets passed to SubQuestionQueryEngine, so it generates an incomplete question: What insurance providers does {patient} have?

Logs and code in thread 🧵. Appreciate any guidance here!

27 comments

JJoshhhh

Logs:

LLogan M

@Joshhhh yes, an agent does rewrite the query

It looks at the latest message + chat history + list of tool names/descriptions, and writes the tool inputs to call a tool (in this case, it writes the query)

JJoshhhh

Here’s my query engine setup:

Plain Text

  top_level_sub_tools = [
      QueryEngineTool(
          query_engine=tool_set["tool_subq"],
          metadata=ToolMetadata(name=tool_set["index_id"], description=tool_set["description"]),
      ) for tool_set in vector_query_engine_tools
  ]
  
  chat_llm = OpenAI(
          temperature=0.1,
          model=OPENAI_CHAT_LLM_NAME,
          streaming=True,
          api_key=OPENAI_API_KEY,
      )
  
  curr_date = datetime.utcnow().strftime("%Y-%m-%d")
  chat_engine = OpenAIAgent.from_tools(
      tools=top_level_sub_tools,
      llm=chat_llm,
      verbose=True,
      system_prompt=SYSTEM_MESSAGE.format(curr_date=curr_date),
      max_function_calls=3,
  )

LLogan M

Maybe a system prompt with some extra instructions would help with how it calls tools

JJoshhhh

Ok, thanks @Logan M

JJoshhhh

Couldn't find out where/how it did that from source code

LLogan M

No worries! With openai, its a bit obfuscated

Basically in the API request, you send a list of chat messages, and a list of tools, and the API handles everything.

Theres no explicit instructions needed, if tools are provided, then the API response may or may not have a selected tool to invoke 👀

JJoshhhh

Changing the SYSTEM_MESSAGE isn't working :/

Even after telling it 3 times to restate the original question

LLogan M

what if instead its in the tool description?

LLogan M

(prompting is hard)

JJoshhhh

Wdym?

LLogan M

Plain Text

top_level_sub_tools = [
    QueryEngineTool(
        query_engine=tool_set["tool_subq"],
        metadata=ToolMetadata(name=tool_set["index_id"], description=tool_set["description"]),
    ) for tool_set in vector_query_engine_tools
]

In the tool description there 👀

LLogan M

I think worst case, if nothing else works, the agent should have some kind of config setting to not use the rewritten tool input?

JJoshhhh

My SUB_QUESTION_PROMPT_TMPL prompt is also instructed to preserve the original user Q, but it's getting transformed before it hits SubQuestionQueryEngine. That prompt is passed via question_gen which is used here: (file)

LLogan M

yea that makes sense. The agent is writing the query before the sub question engine gets invoked

JJoshhhh

Is this the correct setup? Does OpenAIAgent pick the tools or does SubQuestionQueryEngine?

JJoshhhh

Based on SEC Insights, it seems like both do, right? https://github.com/run-llama/sec-insights/blob/eac22605ae43e4b1f9b2904d2bbb015928fb002b/backend/app/chat/engine.py#L257C4-L313C6

LLogan M

They both do! Basically OpenAIAgent is like the frontend for chat. It can respond with or without calling a query engine

JJoshhhh

Got it! Any benefit to having one pick the tool over the other?

JJoshhhh

I had it so that all of the tools were wrapped within SubQuestionQueryEngine last night, but it kept firing questions for all 5 tools. Maybe that's just a prompting issue?

JJoshhhh

(so OpenAIAgent was only picking the 1 SubQuestionQueryEngine)

LLogan M

The openai agent allows for more natural responses to general inputs like Hello! How are you? and keeps track of chat history

The sub question engine doesn't do either of those things

LLogan M

Usuaully thats because its very unsure which query engine to ask questions towards. Can be alleviated with better descriptions of each query engine

JJoshhhh

Got it. Will revert to that approach and will play with the prompts / descriptions

JJoshhhh

Thanks again 🙏

JJoshhhh

Update: Works really well now by having 1 top_level_sub_tool that uses a SubQuestionQueryEngine with many tools. That stopped OpenAIAgent from transforming the original query_str.

LLogan M

perfect!

Add a reply

Find answers from the community

My original user input query is getting