Log in
Log into community
Find answers from the community
View all posts
Related posts
Did this answer your question?
π
π
π
Powered by
Hall
Inactive
Updated 2 months ago
0
Follow
Privacy
Privacy
Inactive
0
Follow
B
Bradathur
last year
Β·
Yea. I personally don't think its an issue but employer is paranoid about that stuff (its understandable)
L
B
5 comments
Share
Open in Discord
L
Logan M
last year
Yea definitely understandable!
You can use llama index with local models, but tbh open source stuff only works well for basic use cases.
Stuff like agents or more complex query engines may not work well π
L
Logan M
last year
Plus then you need the resources to run the local stuff π€
B
Bradathur
last year
yea... i was gonna say probably cannot use a cheap aws instance for that
B
Bradathur
last year
whats the best local model for basic stuff?
L
Logan M
last year
Probably something like the 7B falcon model it seems like. I usually check here
https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard
Anything above 7 to 13B and the requirements start to get a little complex. Although the 4bit quantization helps a lot
Add a reply
Sign up and join the conversation on Discord
Join on Discord