Find answers from the community

Updated 2 months ago

Debugging faithfulness evaluation in rag bot

Trying to replicate Faithfullness eval(https://docs.llamaindex.ai/en/stable/examples/evaluation/faithfulness_eval/) on my RAG bot. But it sometime takes 15 mins, 30 mins, 1hr etc. Anyway to debug it? thanks.
Attachment
image.png
L
J
10 comments
Are you using openai? Some other LLM/embedding model?
yes Llama 3.1
Attachment
image.png
BGE onnx for embedding
Attachment
image.png
Oh boy llamacpp
that will take ages to run yes
Have you considered trying ollama?
Might be a bit faster
but really depends on your hardware
I do have NVIDIA RTX 3060. The prime reason of using LlamaCPP was using a language model stored locally. Since we are building a RAG chatbot that will not be connected to internet at client site, it need to be bundled with all it need to run before shipping out. If i can do that with Ollama, i will never use LlamaCPP again.
do i also need to change embedding?
Add a reply
Sign up and join the conversation on Discord