I am using `RagEvaluatorPack` to

I am using RagEvaluatorPack to evaluate a RAG system I created using the EmbeddedTablesUnstructuredRetrieverPack :

rag_evaluator = RagEvaluatorPack(
    query_engine=query_engine,  
    rag_dataset=rag_dataset,
    )

when I execute
benchmark_df = rag_evaluator.run()
althought I do get some metrics, I run into RuntimeError: Event loop is closed and the script stops

I found the error usually occurs because of Asynchronous jobs, but I am not sure

I'd appreciate your insight/ help if anyone has an idea

3 comments

LLogan M

usually that happens because of errors further up? What there any other errors?

NNeza

here's the whole log :

FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.

  ## these are the results
mean_scores_df = pd.concat(
rag                            base_rag
metrics
mean_correctness_score         3.500000
mean_relevancy_score           0.470588
mean_faithfulness_score        0.882353
mean_context_similarity_score       NaN

## then here, it stopped
Exception ignored in: <function _ProactorBasePipeTransport.__del__ at 0x0000021EACB2AA70>
Traceback (most recent call last):
  File "C:\Users\qwerty\AppData\Local\Programs\Python\Python310\lib\asyncio\proactor_events.py", line 116, in __del__   
  File "C:\Users\qwerty\AppData\Local\Programs\Python\Python310\lib\asyncio\proactor_events.py", line 108, in close     
  File "C:\Users\qwerty\AppData\Local\Programs\Python\Python310\lib\asyncio\base_events.py", line 745, in call_soon     
  File "C:\Users\qwerty\AppData\Local\Programs\Python\Python310\lib\asyncio\base_events.py", line 510, in _check_closed 
RuntimeError: Event loop is closed

it seems like it's stemming from asyncio ?

LLogan M

Does the context similarity score being NaN make sense?

(Asyncio like to swallow/hide real errors sadly)

Add a reply

Find answers from the community

I am using `RagEvaluatorPack` to