Find answers from the community

Updated 8 months ago

need help with BatchEvalRunner

need help with BatchEvalRunner
i tried to follow this example: https://docs.llamaindex.ai/en/stable/examples/evaluation/batch_eval/

this is my code
Plain Text
print("*** start evaluation")
    faithfulness = FaithfulnessEvaluator(service_context=service_context)
    relevancy = RelevancyEvaluator(service_context=service_context)
    correctness = CorrectnessEvaluator(service_context=service_context,score_threshold=2.0 ,parser_function=eval_parser)

    runner = BatchEvalRunner(
                                { "faithfulness": faithfulness, "relevancy": relevancy},
                                    workers=8, show_progress=True
                            )
    engine = index.as_query_engine()
    
    eval_results = await runner.aevaluate_queries( engine, queries=eval_questions.questions )


for some reason, i am getting the error below
Plain Text
Traceback (most recent call last):
  File "/Users/ilanpinto/dev/ai/lightspeed-rag-content/.conda/lib/python3.11/site-packages/llama_index/core/evaluation/batch_runner.py", line 268, in aevaluate_responses
    multi_kwargs = isinstance(next(iter(eval_kwargs_lists.values())), dict)
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
StopIteration
L
i
d
18 comments
Hmm, seems like a bug. I'll take a look. Try downgrading llama-index-core for now
to which version?
i have downgraded to 0.10.17 and got the same errir
are you running in a notebook? You might have to restart the runtime/notebook to properly update
running on vscode
getting the same error without upgrading. been on 0.10.24 and this error showed up today
@ilan_pinto @drtuc just merged a fix into main. Try pip install git+https://github.com/run-llama/llama_index.git -- it worked for me at least
still getting the same :

Traceback (most recent call last):
File "/Users/ilanpinto/dev/ai/rag-pipeline/rag/lib/python3.11/site-packages/llama_index/core/evaluation/batch_runner.py", line 268, in aevaluate_responses
multi_kwargs = isinstance(next(iter(eval_kwargs_lists.values())), dict)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
seems like pip install git+https://github.com/run-llama/llama_index.git doesn't downloads the latest version
i had to apply it manuly
uhhh it should, but maybe its pip install -U git+...
When are you planning to upgrade version with this fix?
probably today my guy (releases are a PITA)
(installing from source worked fine for me, fyi)
I am not sure why it's not working for me. seems like it's cached somewhere
fyi its out now as v0.10.27 btw
Add a reply
Sign up and join the conversation on Discord