Hi all, new here but really excited

MMike

Hi all, new here but really excited about the project. Wondering if I can get some help with the following:
When querying an index with index.query_as_engine with an output_cls specified, I'm getting a pydantic validation error when the output doesn't match my output class. Is there something I can do to prevent or remedy this?

11 comments

LLogan M

are you using an open source LLM?

MMike

I am not. gpt-4-1106-preview until I reached the TPD. And now gpt-4

LLogan M

interesting. I guess the LLM is just not predicting the output class properly (either it's running out of tokens to predict becasue the class object is large, or the output class is complex)

MMike

Definitely happened less often on gpt-4 turbo I should mention. Hmm yeah the output class is fairly large compared to the examples, I'll try to simplify it. In regards to it potentially running out of tokens, is there an easy way to see how many tokens are included in the relevant calls? I didn't see it with the debug logs on or setting verbose=True but it's possible I missed it.

LLogan M

hmm, not an easy way to see (especially since token counting for tools is a mystery). When it fails though, does it end up printing the json it failed to parse or no?

MMike

The fields that are failing are enums. So the error message shows the returned value from the model and the list of acceptable values for that field.

E.g.

Plain Text

...
 line 204, in structured_predict
    return program(**prompt_args)
  File "/Users/[username]/.pyenv/versions/3.9.15/lib/python3.9/site-packages/llama_index/program/openai_program.py", line 185, in __call__
    return _parse_tool_calls(
  File "/Users/[username]/.pyenv/versions/3.9.15/lib/python3.9/site-packages/llama_index/program/openai_program.py", line 63, in _parse_tool_calls
    output = output_cls.parse_raw(function_call.arguments)
  File "/Users/[username]/.pyenv/versions/3.9.15/lib/python3.9/site-packages/typing_extensions.py", line 2499, in wrapper
    return arg(*args, **kwargs)
  File "/Users/[username]/.pyenv/versions/3.9.15/lib/python3.9/site-packages/pydantic/main.py", line 986, in parse_raw
    return cls.model_validate(obj)
  File "/Users/[username]/.pyenv/versions/3.9.15/lib/python3.9/site-packages/pydantic/main.py", line 480, in model_validate
    return cls.__pydantic_validator__.validate_python(
pydantic_core._pydantic_core.ValidationError: 1 validation error for CrashReport
manner_of_collision
  Input should be 'Rear End','Not a Collision with Motor Vehicle','Angle (Other)','Sideswipe-Same Direction','Left Angle Crash','Head On','Sideswipe-Opposite Direction','Right Angle Crash' or '(None)' [type=enum, input_value='Guard Rail Face', input_type=str]

LLogan M

ahhh enums -- tbh I've never been able to get enums to work in an output_cls

LLogan M

I'm not sure if its a bug with how we convert the class to a json representation to the API, or if the LLM just wasn't trained with them, or what the deal is

LLogan M

haven't dug too much into it, but yea

MMike

Interesting. I appreciate your help quite a bit! Simplifying the class a little bit and providing better descriptions has helped already so I might be okay in the short term. Longer term it feels like with the modules llama-index exposes I could potentially customize behavior around this flow to resubmit the error and the input to the model and ask it to fix it. Might be a naive solution, but I'm sure with how fast things are moving won't be a problem for long.

MMike

Thanks again!

Add a reply

Find answers from the community

Hi all, new here but really excited