For now I think only LLMs can return structured outputs. If I wanted to get an agent to return a structured output, my only way of doing so would be to use tool = FunctionTool.from_defaults(fn={fn}, return_direct=True) right?
Except that I need to be mindful to ensure that my tool output is a string so that I won't mess up the AgentChatResponse output class. So if I specify a pydantic base model as my desired return, I would have to store the key value pairs of the response model as a dictionary then json dumps that dictionary as my return, and then add postprocessing code to handle the return.
Of course the other way would be to write a custom agent using Workflow and handle the ToolOutput there.
Personally I would take the workflow approach. There is always going to be more that you want control over or need to debug, and the prebuilt agents are little too black box imo
