For now I think only LLMs can return

For now I think only LLMs can return structured outputs. If I wanted to get an agent to return a structured output, my only way of doing so would be to use tool = FunctionTool.from_defaults(fn={fn}, return_direct=True) right?

Except that I need to be mindful to ensure that my tool output is a string so that I won't mess up the AgentChatResponse output class. So if I specify a pydantic base model as my desired return, I would have to store the key value pairs of the response model as a dictionary then json dumps that dictionary as my return, and then add postprocessing code to handle the return.

Of course the other way would be to write a custom agent using Workflow and handle the ToolOutput there.