Find answers from the community

Updated 2 months ago

Hello. I am trying to evaluate a

Hello. I am trying to evaluate a retriever, and I see that using the methods in this documentation: https://docs.llamaindex.ai/en/stable/examples/evaluation/retrieval/retriever_eval.html , we generate question context pairs. Say that I already have a question context pair, how can I use LlamaIndex's RetrieverEvaluator to fit with my generated question context pair?
L
o
2 comments
You'll need the text and id of the expected node(s) from the query

Plain Text
eval_result = retriever_evaluator.evaluate(sample_query, [expected_id1], [expected_text1])
print(eval_result)
I'd maybe want to test out different chunking methods, so I may not know the expected node ids in advance. Would it be possible to use only expected_text?

Or maybe have the expected id be the id of the document, so that if a node comes from a certain document, it is considered a hit.
Add a reply
Sign up and join the conversation on Discord