Find answers from the community

Updated 2 months ago

ollama/docs/modelfile.md at main · ollam...

Odd issue with settings ollama kwargs. Looking at the ollama documentation for the possible additional arguments:
https://github.com/ollama/ollama/blob/main/docs/modelfile.md

For num_predict - it tells you the default is 128. However if you dont set this in the additional arguments variable in llamaindex, you get way more than 128 as a response.

However if you set num_predict = 128 as an addition kv arg in llamaindex, it several limits the context of the response. It is easy enough to set this, but I am confused on what this value actually is if you dont set it.
L
i
3 comments
Ollama might be setting it to -1 by default? Thats just the default in the modelfile

The actualy Ollama llm class is not setting this var
Yeah in that openai go file, it gets the value of max_tokens if left unset.
Which would make sense. -1 is infinite. But I don't fully understand the implications of this vs max_tokens
Add a reply
Sign up and join the conversation on Discord