Find answers from the community

Updated 4 months ago

Eval

At a glance

Hi guys I am using this

Plain Text

        retriever_evaluator = RetrieverEvaluator.from_metric_names(
            ["mrr", "hit_rate"], retriever=custom_retriever
        )
        eval_results = retriever_evaluator.evaluate_dataset(qa_dataset)

AttributeError: 'RetrieverEvaluator' object has no attribute 'evaluate_dataset'
how to resolve this

5 comments

LLogan M

I think you need to use async?

https://github.com/run-llama/llama_index/blob/4e8d3d8cabfea5a2289c01fb9d435a50cdb8c360/llama-index-core/llama_index/core/evaluation/retrieval/base.py#L172

LLogan M

aevalute_dataset(...)

rrishi

Thanks

rrishi

Hi
Is it possible to calculate
First for all the chunks in the document, call embedding model and calculate the embeddings, and store them in a dict / list
Second for each query, calculate its embedding
Third for each query do cosine sim with all the emebdding of the document
and while doing all of that can we also calculate the embedding time for each query and the embedding time for the whole dataset

rrishi

@Logan M

Add a reply