rishi

Log inLog into community

Find answers from the community

Home

Members

rishi

Offline, last seen 3 months ago

Joined September 25, 2024

rrishi

Eval

Hi guys I am using this

Plain Text

        retriever_evaluator = RetrieverEvaluator.from_metric_names(
            ["mrr", "hit_rate"], retriever=custom_retriever
        )
        eval_results = retriever_evaluator.evaluate_dataset(qa_dataset)

AttributeError: 'RetrieverEvaluator' object has no attribute 'evaluate_dataset'
how to resolve this

5 comments

rrishi

I want to figure out the time it takes to embed a whole dataset and the time it takes to e

@kapa.ai I want to figure out the time it takes to embed a whole dataset and the time it takes to embed a single query and later on find the time it takes to embed a single query in a list of queries . How to do this

31 comments

rrishi

Hi

Hi
Is it possible to calculate
First for all the chunks in the document, call embedding model and calculate the embeddings, and store them in a dict / list
Second for each query, calculate its embedding
Third for each query do cosine sim with all the emebdding of the document
and while doing all of that can we also calculate the embedding time for each query and the embedding time for the whole dataset

20 comments