LlamaIndex
Log in

Find answers from the community

Updated 2 years ago

Has anyone experienced local Llama cpp

At a glance
AAdrian
·
Has anyone experienced local Llama.cpp models omitting words? The models run great when I run the llama.cpp models via ./server. However, when I use them in LlamaIndex I get responses with omissions. I'm using Llama-2-13B-chat.
L
A
D
11 comments
LLogan M
I've seen a few issues about this on the llama-cpp-python github, but haven't read much more into it, might be good to search there
AAdrian
Nice, thank you @Logan M , I'll check out the repo.
AAdrian
@Logan M Genius. https://github.com/abetlen/llama-cpp-python/pull/644
AAdrian
Thank you, dear friend.
LLogan M
Nice!
LLogan M
Heads up though if you update your llama-cpp version
LLogan M
They stopped supporting GGML files, only GGUF from now on
LLogan M
need to update the default file thats downloaded in llama-index, but GGUF files aren't too common yet 😅
AAdrian
Yeah, I've been converting all my GGML files to GGUF. Good looking out.
DDangFutures
do you know if gguff run on windows?
LLogan M
It should! I personally haven't tried though
Add a reply
Sign up and join the conversation on Discord