Find answers from the community

Updated 9 months ago

We're getting a lot of Pydantic

We're getting a lot of Pydantic validation errors in our prod environment for queries using GPT-3-turbo. Is this a common thing? Is there anything we can do to make sure this happens less? I would assume llama index uses JSON mode for OpenAI under the hood?

11 comments

LLogan M

its actually not using JSON mode, but it is use function/tool calling in 90% of cases

LLogan M

Although even that isn't 100% reliable, especially with 3.5

LLogan M

I've seen it call tools will hallucinated or missing fields

LLogan M

If its a pydantic error, that means it already created a parseable JSON object

NNiels

Is there any way to improve the reliability of this? @Logan M

Maybe a retry mechanism or something? We are seeing this at least 100+ times a day in our production environment

LLogan M

How are you using this?

LLogan M

Like, what component/setup?

NNiels

We are using output_cls where we pass the Pydantic models:

Plain Text

 query_engine = index.as_query_engine(
   llm=llm, output_cls=PresentationContentList, similarity_top_k=2, filters=filters,
 )

LLogan M

Hmm, my best guess to improve this would be to either try/except the query and retry

Or, add some code here to retry on this pydantic error
https://github.com/run-llama/llama_index/blob/f513d6c21be76b0a626d7f35b93bbb1420afba36/llama-index-integrations/program/llama-index-program-openai/llama_index/program/openai/base.py#L159

LLogan M

I'm not sure what you set your temperature to, but something other than zero may help

NNiels

Thanks a lot, will give it a try

Add a reply