Privacy

Yea definitely understandable!

You can use llama index with local models, but tbh open source stuff only works well for basic use cases.

Stuff like agents or more complex query engines may not work well 😅

Plus then you need the resources to run the local stuff 🤔

yea... i was gonna say probably cannot use a cheap aws instance for that

whats the best local model for basic stuff?

Probably something like the 7B falcon model it seems like. I usually check here

https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard

Anything above 7 to 13B and the requirements start to get a little complex. Although the 4bit quantization helps a lot