Find answers from the community

Updated 5 months ago

I recently updated llama-index-llms-

I recently updated llama-index-llms-openai 0.1.26 to use 4o-mini. On openai's website they said CONTEXT WINDOW 128k tokens but when I tried to set the limit it said

Plain Text

Error code: 400 - {\'error\': {\'message\': \'max_tokens is too large: 120000. This model supports at most 16384 completion tokens, whereas you provided 120000.\', \'type\': \'invalid_request_error\', \'param\': \'max_tokens\', \'code\': None}}\n'}

8 comments

WWhiteFang_Jr

Yes the context window is 128K only: https://github.com/run-llama/llama_index/blob/6a8e151f9b912d8fad5fa4d09bd2f7bfcb393f0c/llama-index-integrations/llms/llama-index-llms-openai/llama_index/llms/openai/utils.py#L50

Can you double check if you are using GPT-4o mini only

DDat Tran

Interesting, I have a different part of the code using 3.5 for SQLTableRetrieverQueryEngine. Didn't know they interfere with eachother.

DDat Tran

A month back I asked about gpt-4 and sql query. 3.5 will follow the instruction and do query but gpt-4 will refuse to query directly to db. Has anything changed since then?

WWhiteFang_Jr

You'll have to make the prompt more clean and more descriptive. Try with GPT-4o or mini see if it works for you

GGeoloeG

oh, LO @dev_advocate nice to see you around! 😄

rrajiv

Many thanks @WhiteFang_Jr ! Could you say more or share some good+bad examples of a "more clean and more descriptive" prompt for optimal reliability with gpt-4o mini and SQL/db queries?

WWhiteFang_Jr

You can checkout llama-index default prompts. This will give you an idea for how to define more descriptive and detailed prompt:https://github.com/run-llama/llama_index/blob/34dec27b7df959e05f9aa5a5859fb4c1d10e2b21/llama-index-core/llama_index/core/prompts/default_prompts.py#L188

rrajiv

thanks!

Add a reply