Enabling Structured Output for FunctionCallingAgents Wi...

At a glance

The community members are discussing how to enable structured output for FunctionCallingAgents without additional cost. The main suggestions are:

1. Declare a FunctionTool with return_direct=True and a tool name/description that signals the LLM to invoke it for the final response. This may also require a system prompt.

2. Create a custom finalize_response function that returns a dictionary with the desired structured output fields (e.g., "answer" and "justification"). Use this function in a FunctionTool with return_direct=True, and set the system prompt to always use this tool last.

The community members agree that the second approach is simpler and should work well, even if it doesn't use the native structured output support from the language models.

zzhaoyilong1

Hi folks, is there a way to enable structured output for FunctionCallingAgents (not ReAct agent) without additional cost (just wrap/use structured llm in some way, or call structured_predict, instead of using another agent to process response).

10 comments

LLogan M

I think the easiest way is to declare a FunctionTool with return_direct=True -- that tool would have to have a tool name+description that signals to the LLM it should invoke this to return a final response (probably also need a system prompt to help here)

LLogan M

structured llm doesn't help here, since the agent still needs to be able to call other tools before its final response

zzhaoyilong1

thanks Logan, I think it makes sense, so we need a function tool which will use a structured llm / structured_predict to do this? together with maybe other changes inside the agent.

LLogan M

Actually, I think its even easier. Let me make an example

Suppose you want the structured output to have an answer and justifcation field:

LLogan M

Plain Text

def finalize_response(answer: str, justification: str) -> str:
  """Used to finalize a response to the user. Should be called last once you have any kind of answer for the user."""
  return {"answer": answer, "justification": justification}

final_tool = FunctionTool.from_defaults(fn=finalize_response, return_direct=True)

agent = FunctionCallingAgent.from_tools([..., final_tool], system_prompt="Always use the finalize_response tool, and always use it last.")

LLogan M

something like that is really all you need

zzhaoyilong1

got it, thanks a lot!

zzhaoyilong1

was trying to use the native structured output support from the llms, but this probably should be good enough

LLogan M

The agent could chose not to invoke this response, in that case I would detect that and create a dummy structured response

Something like

Plain Text

resp = agent.chat(...)
try:
  structured_resp = json.loads(str(resp))
except:
  structured_resp = {"answer": str(resp), "justification": None}

LLogan M

should work pretty well

Add a reply

Find answers from the community

Enabling Structured Output for FunctionCallingAgents Without Additional Cost