Hi, I'm trying to use `as_structured

At a glance

The community member is trying to use the as_structured_llm with llama_index.llms.openai.OpenAI and pass it to llama_index.agent.openai.OpenAIAgent.from_tools(), but is getting a ValueError because the as_structured_llm changes the LLM enough that it's no longer considered an OpenAI instance.

The community members discuss potential workarounds, such as using a function calling agent, but they seem to encounter issues with the incompatibility between the structured LLM and the agent's tool calling. They explore various approaches, including using a non-structured LLM to get the initial response and then using a structured LLM to format the output, as well as creating a custom agent. The community members conclude that a custom workflow is the most viable option, as it provides more flexibility and control, although it requires slightly more code.

There is no explicitly marked answer in the post and comments.

Useful resources

UU1F40E

Hi, I'm trying to use as_structured_llm with llama_index.llms.openai.OpenAI and pass that as the llm to llama_index.agent.openai.OpenAIAgent.from_tools(), but I'm getting an error ValueError: llm must be a OpenAI instance. It seems as_structured_llm changes the LLM enough that it's not considered an OpenAI instance anymore.

Is there a better way to produce structured output from an OpenAIAgent with tools?

20 comments

LLogan M

hmm, seems like a bug, workaround would be just using a function calling agent

Plain Text

from llama_index.core.agent import FunctionCallingAgentWorker

agent = FunctionCallingAgentWorker.from_tools(...),as_agent()

UU1F40E

Thinking about this some more, maybe as_structured_llm will prevent the LLM from invoking tools?

UU1F40E

I'll try this. Thanks

LLogan M

oh hmm, good point actually. It might actually be breaking the agents tool calling too 😅

UU1F40E

Logically, I'm not sure how to accomplish both. 🤔

LLogan M

Me neither actually. Seems like theres a general incompatibility with how the structured llm works and how agents work

UU1F40E

The API rejects this as well. AttributeError: 'StructuredLLM' object has no attribute 'achat_with_tools'

UU1F40E

Maybe I can ask the non-structured LLM to answer the prompt with sufficient context and then run a follow-up with the structured LLM to format it into the needed structure.

UU1F40E

For future reference, this kinda works:

Plain Text

await agent.achat(question.prompt)

messages = [ 
    agent.memory.get_all()[-1],
    ChatMessage.from_str("Please convert your previous message into the required format."),
]

response = await structured_llm.achat(messages=messages)

Where agent has the tool(s) and structured_llm has the structure. The parsed object is in response.raw.

UU1F40E

I thought I was clever and tried passing a tool for the output. This is similar to how StructuredLLM does it, but I'm including my other tools as well. Unfortunately, the AgentRunner forces tool_choice to "auto" after the first step, and the LLM doesn't call my output function. 😦 🤔