need help with BatchEvalRunner

iilan_pinto

need help with BatchEvalRunner
i tried to follow this example: https://docs.llamaindex.ai/en/stable/examples/evaluation/batch_eval/

this is my code

Plain Text

print("*** start evaluation")
    faithfulness = FaithfulnessEvaluator(service_context=service_context)
    relevancy = RelevancyEvaluator(service_context=service_context)
    correctness = CorrectnessEvaluator(service_context=service_context,score_threshold=2.0 ,parser_function=eval_parser)

    runner = BatchEvalRunner(
                                { "faithfulness": faithfulness, "relevancy": relevancy},
                                    workers=8, show_progress=True
                            )
    engine = index.as_query_engine()
    
    eval_results = await runner.aevaluate_queries( engine, queries=eval_questions.questions )

for some reason, i am getting the error below

Plain Text

Traceback (most recent call last):
  File "/Users/ilanpinto/dev/ai/lightspeed-rag-content/.conda/lib/python3.11/site-packages/llama_index/core/evaluation/batch_runner.py", line 268, in aevaluate_responses
    multi_kwargs = isinstance(next(iter(eval_kwargs_lists.values())), dict)
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
StopIteration

18 comments

LLogan M

Hmm, seems like a bug. I'll take a look. Try downgrading llama-index-core for now

iilan_pinto

to which version?
i have downgraded to 0.10.17 and got the same errir

iilan_pinto

error

LLogan M

are you running in a notebook? You might have to restart the runtime/notebook to properly update

iilan_pinto

running on vscode

ddrtuc

getting the same error without upgrading. been on 0.10.24 and this error showed up today

LLogan M

@ilan_pinto @drtuc just merged a fix into main. Try pip install git+https://github.com/run-llama/llama_index.git -- it worked for me at least

iilan_pinto

still getting the same :

Traceback (most recent call last):
File "/Users/ilanpinto/dev/ai/rag-pipeline/rag/lib/python3.11/site-packages/llama_index/core/evaluation/batch_runner.py", line 268, in aevaluate_responses
multi_kwargs = isinstance(next(iter(eval_kwargs_lists.values())), dict)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^