Log in
Log into community
Find answers from the community
View all posts
Related posts
Did this answer your question?
😞
😐
😃
Powered by
Hall
Inactive
Updated last year
0
Follow
Anyone else having issues loading the
Anyone else having issues loading the
Inactive
0
Follow
c
captam_morgan
last year
·
Anyone else having issues loading the 70B llama2 model on LlamaCPP? I was successful with the 7B and 13B models but I’m getting a vague error for 70B. (See attached image)
My cluster is CPU only but has up to 96 workers and 768GB ram.
Attachments
L
c
11 comments
Share
Open in Discord
L
Logan M
last year
Is there anything useful in that assertion error? Looks like it's cutoff
L
Logan M
last year
oh wait nvm haha
L
Logan M
last year
I see
L
Logan M
last year
Is the model path correct?
L
Logan M
last year
Seems like somehow the model is not being loaded 🤔
c
captam_morgan
last year
I removed the real path in the photo but I used the right path when I ran into that error
c
captam_morgan
last year
It’s the same dir as the other small models
c
captam_morgan
last year
Let me debug some more before you spend any time on it, you’ve been super human in terms of answering all out questions lol
L
Logan M
last year
haha sounds good! Let me know if you find any clues!
I actually haven't tried to run the 70B model with llama.cpp yet, not enough RAM on my end 😉
c
captam_morgan
last year
Found the solution! The bloke himself said to set "n_gqa": 8 for the 70B model. And it worked!
Attachment
L
Logan M
last year
That's so random hahaha nice!
Add a reply
Sign up and join the conversation on Discord
Join on Discord
