Find answers from the community

Updated last year

Hardware

Hi! I need some help regarding hardware for running an llm offline... where is the best place to ask questions about that?
E
d
3 comments
you can send the questions on this thread, then we can see how we can help you
Sure! Thank you for creating the thread. So I am unsure of what hardware I would need to run a Llama 2 70b model. I know I need at least 64gb of RAM just to run the quantized version. My confusion stems from me trying llms on macos where knowing how much ram I need is easy as its just the "unified memory". For a desktop, do I need the required amount of RAM in regular DDR5 RAM or in my graphics card VRAM?
I am also open for recommendations on CPU and motherboard options as well
Add a reply
Sign up and join the conversation on Discord