Hi, I'm trying to use `as_structured

UU1F40E

Hi, I'm trying to use as_structured_llm with llama_index.llms.openai.OpenAI and pass that as the llm to llama_index.agent.openai.OpenAIAgent.from_tools(), but I'm getting an error ValueError: llm must be a OpenAI instance. It seems as_structured_llm changes the LLM enough that it's not considered an OpenAI instance anymore.

Is there a better way to produce structured output from an OpenAIAgent with tools?

20 comments

LLogan M

hmm, seems like a bug, workaround would be just using a function calling agent

Plain Text

from llama_index.core.agent import FunctionCallingAgentWorker

agent = FunctionCallingAgentWorker.from_tools(...),as_agent()

UU1F40E

Thinking about this some more, maybe as_structured_llm will prevent the LLM from invoking tools?

UU1F40E

I'll try this. Thanks

LLogan M

oh hmm, good point actually. It might actually be breaking the agents tool calling too 😅

UU1F40E

Logically, I'm not sure how to accomplish both. 🤔

LLogan M

Me neither actually. Seems like theres a general incompatibility with how the structured llm works and how agents work

UU1F40E

The API rejects this as well. AttributeError: 'StructuredLLM' object has no attribute 'achat_with_tools'

UU1F40E

Maybe I can ask the non-structured LLM to answer the prompt with sufficient context and then run a follow-up with the structured LLM to format it into the needed structure.

UU1F40E

For future reference, this kinda works:

Plain Text

await agent.achat(question.prompt)

messages = [ 
    agent.memory.get_all()[-1],
    ChatMessage.from_str("Please convert your previous message into the required format."),
]

response = await structured_llm.achat(messages=messages)

Where agent has the tool(s) and structured_llm has the structure. The parsed object is in response.raw.

UU1F40E

I thought I was clever and tried passing a tool for the output. This is similar to how StructuredLLM does it, but I'm including my other tools as well. Unfortunately, the AgentRunner forces tool_choice to "auto" after the first step, and the LLM doesn't call my output function. 😦 🤔