Hi, I'm following the example of the

Not sure If I'm right but I have seen this pattern 😅

AVX = 1 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 |

And maybe Logan had mentioned that your model is going out of memory while loading.

MAYBE!

strange. I'm running on CPU with 64GB or RAM. I've seen a message related to the same ouput error message that suggested tot use a gguf instead of a ggmlv3...

I'll try with this model:
model_url = "https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGUF/resolve/main/llama-2-7b-chat.Q2_K.gguf"

Yeah could be. That's why i added MAYBE 😅

Found one thread:https://discord.com/channels/1059199217496772688/1174774402919960642/1174782379781529720

thanks for the indication 🙂
it's working with the gguf format

inference time is decent given that I'm using a CPU 1165g7 intel

yeah it's actually good 👍

do you know if it's possible to install llamacpp in another laptop with M1 Pro CPU and make the call from the less powerful laptop with Intel CPU?
I could do something similar with Ollama but in the section of Ollama I don't see that llamaindex has anything similar. With ollama one uses the requests paackage like this:

url = "http://192.168.1.xyz:11435/api/generate" data = { "model": 'llama2-uncensored', "prompt": prompt, "stream": False } result = requests.post(url, json=data) json_data = json.loads(result.text)

It's there, When you are defining llm via Ollama, Pass the base_url in the Ollama as well.

https://github.com/run-llama/llama_index/blob/43bb20c4ffeed1644e13700a564d6cbcfb726951/llama_index/llms/ollama.py#L32

thanks for the help @WhiteFang_Jr
for a 7B parameter model there was significative gain changing the CPU

This is way better @davidp 🎉

I'm trying to make the whole RAG thing with local resources. For the embeddings, Ollama doesn't have that funcionality. So, I've used langchain to load a model from huggingface. It's working pretty well, but do you know if its a good idea to mix embeddings from a model X (lbge-base-en) and then use a model Y(llama2 7B) to make the generation of the final answer?

Yes its fine. As embedding model works separately from the llm.

It converts all the docs into embedding and then retrieve the related chunks based on user query and then it is passed to the LLM.


Also, You can also use HF embedding from llamaindex also: https://docs.llamaindex.ai/en/stable/examples/embeddings/huggingface.html#huggingfaceembedding

thanks. It also works fine

actually, I was wondering what is the final query passed to the ollama model from llamaindex at the generation stage. Theoretically, the retrieval gets some documents or chunks of documents but what is then told to ollama? something like "make me a summary of the following chunks:". Is it possible to see the prompt engineering in behind?
before these tests with llamaindex, I was using weaviate with gpt4all embeddings and for the generation stage I was doing the call to ollama with a prompt telling to make a summary of all the chunks retrieved.

I think this can help you in tracing the calls to LLM and more:https://docs.llamaindex.ai/en/stable/module_guides/observability/observability.html#arize-phoenix

Hi @WhiteFang_Jr , I decided to use the Traceloop to see the constructed prompt that is passed to the LLM based on the query and the retrieved documents. The Traceloop team had to update their library to support Ollama but it's now working.
My concern is if the constructed prompt can also be seen from the command line with some option for the query engine or some other class...

at least the prompt shape can be obtained like this: