Find answers from the community

Updated 10 months ago

Hi guys, sorry quick question, is there

Hi guys, sorry quick question, is there a way to set a global top_k, top_p, max_tokens in Settings.llm? I'm using Mistral and couldn't find anything in the documentation (generate_kwarg and additional_kwarg seems not to work)
W
m
5 comments
top_p and max_tokens are part of LLM , setting it while initialising your LLM makes more sense.

For top_K, you can set it in the query_engine directly!.

Are they not working when you set it there?
Thanks for your reply!

It's ok for the top_k,

I agree with you about setting top_p and max_tokens while initialising my LLM that's why I would like to use it like this :

"Settings.llm = MistralAI(api_key=api_key, model="mistral-small", top_p = 0.90, max_tokens = 300) "

There is another way ?
keyword should be exact: additional_kwargs
Thanks a lot !
Add a reply
Sign up and join the conversation on Discord