Find answers from the community

Updated 5 months ago

hello, i want to work fully locally

At a glance
hello, i want to work fully locally without any connexion to internet. i aim to download the llm from HF ( text_to_sql project) to bypass the need to the key but i dont know how to replace :Settings.embed_model = HuggingFaceEmbedding(
model_name="BAAI/bge-small-en-v1.5"
) and llm = HuggingFaceInferenceAPI(
model_name="HuggingFaceH4/zephyr-7b-alpha", token=HF_TOKEN
). any help plzzzz
W
m
5 comments
If you are going to use HF inference API it will require internet
You can setup HF model locally but this will require good RAM and GPU to run properly.
https://docs.llamaindex.ai/en/latest/examples/llm/huggingface/

You can also check Ollama, its pretty easy to setup and they have optimized the model size in there.
https://docs.llamaindex.ai/en/latest/examples/llm/ollama_gemma/
i downloaded all the file of the llm in my machine and i want to pypass the call to HF or Ollama because both of them requires internet. ( even ollama i think ) so i don't know how to "call" the model from my machine
Are you running Ollama instance somewhere else? apart from your local machine?
no ! but maybe i can use llamaCpp and specify the path to the model? for the embedding i'm not sure
Add a reply
Sign up and join the conversation on Discord