Find answers from the community

Updated 9 months ago

We're getting a lot of Pydantic

We're getting a lot of Pydantic validation errors in our prod environment for queries using GPT-3-turbo. Is this a common thing? Is there anything we can do to make sure this happens less? I would assume llama index uses JSON mode for OpenAI under the hood?
L
N
11 comments
its actually not using JSON mode, but it is use function/tool calling in 90% of cases
Although even that isn't 100% reliable, especially with 3.5
I've seen it call tools will hallucinated or missing fields
If its a pydantic error, that means it already created a parseable JSON object
Is there any way to improve the reliability of this? @Logan M

Maybe a retry mechanism or something? We are seeing this at least 100+ times a day in our production environment
How are you using this?
Like, what component/setup?
We are using output_cls where we pass the Pydantic models:

Plain Text
 query_engine = index.as_query_engine(
   llm=llm, output_cls=PresentationContentList, similarity_top_k=2, filters=filters,
 )
Hmm, my best guess to improve this would be to either try/except the query and retry

Or, add some code here to retry on this pydantic error
https://github.com/run-llama/llama_index/blob/f513d6c21be76b0a626d7f35b93bbb1420afba36/llama-index-integrations/program/llama-index-program-openai/llama_index/program/openai/base.py#L159
I'm not sure what you set your temperature to, but something other than zero may help
Thanks a lot, will give it a try
Add a reply
Sign up and join the conversation on Discord