Hi,
So basically I am using a different approach, my RAG pipe line model code is;
summarizer = TreeSummarize(
service_context = ServiceContext.from_defaults(
llm=llm, embed_model=embed_model,chunk_size=1024
)
)
service_context = ServiceContext.from_defaults(chunk_size=1024, llm=llm, embed_model=embed_model)
index = VectorStoreIndex.from_documents(pdfdocuments,service_context=service_context)
retriever = index.as_retriever(similarity_top_k=3)
p = QueryPipeline(verbose=True)
p.add_modules(
{
"input": InputComponent(),
"retriever": retriever,
"summarizer": summarizer,
}
)
p.add_link("input", "retriever")
p.add_link("input", "summarizer", dest_key="query_str")
p.add_link("retriever", "summarizer", dest_key="nodes")
output = p.run(input='How is the current economic and political climate expected to impact consumer behavior during Ramazan 2023?')
output_str = output.response
I want to evaluate this RAG pipeline, I was trying RAGAS and trulens both of them giving me the import error, i tried different solution but none of them work. I posted my problem in this channel and get the link from Jack which was this
https://docs.llamaindex.ai/en/stable/examples/evaluation/retrieval/retriever_eval/. When going through the link I use this code to get the metric
from llama_index.core.evaluation import RetrieverEvaluator
metrics = ["hit_rate", "mrr", "precision", "recall", "ap", "ndcg"]
retriever_evaluator = RetrieverEvaluator.from_metric_names(
metrics, retriever=retriever)
and got this error ;
ValueError: Invalid metric name: precision
Also I am want to know how to get the expected id from the service context I have used?