Debugging faithfulness evaluation in rag bot

Are you using openai? Some other LLM/embedding model?

yes Llama 3.1

BGE onnx for embedding

Oh boy llamacpp

that will take ages to run yes

Have you considered trying ollama?

Might be a bit faster

but really depends on your hardware

I do have NVIDIA RTX 3060. The prime reason of using LlamaCPP was using a language model stored locally. Since we are building a RAG chatbot that will not be connected to internet at client site, it need to be bundled with all it need to run before shipping out. If i can do that with Ollama, i will never use LlamaCPP again.

do i also need to change embedding?