When using Query Pipelines is there a

curious what you mean by "not consistent" ?

You could have a query pipeline component to parse the LLM output and/or correct the JSON?

Yes, I am doing exactly that (correcting the JSON in a pipeline component) and it works well for gpt-3.5, but for gpt-4 I keep getting messages before and after the JSON that my script can't handle. I can improve my script for cleaning up the JSON output, but I just wanted to know if there was a way to call for gpt-4's JSON output feature from the pipeline.

Have you tried just using the tool calling api instead? (i.e OpenAIPydanticProgram?)

I think you can activate json mode by setting it in the constructor too
llm = OpenAI(..., additional_kwargs={"response_format": {"type": "json_object"}})

Theres actually a comparison here too (they pass in the response format into the chat call, additional_kwargs works too)
https://docs.llamaindex.ai/en/stable/examples/llm/openai_json_vs_function_calling/?h=json

Yes, I saw that tutorial, but I didn't realize that I can set JSON mode in the constructor. That looks promising and I am going to give that a try. Thank you!

I'm getting this error when using gpt-4:
"Invalid parameter: 'response_format' of type 'json_object' is not supported with this model."
I'll see what happens with gpt-3.5...

I wonder if they changed the param name? Thats kind of odd

Seems like it might only be specific models yea

Yeah it worked for 3.5 turbo. Trying 4 turbo now. Thanks.