Find answers from the community

Updated last year

https://docs.llamaindex.ai/en/stable/

At a glance

https://docs.llamaindex.ai/en/stable/module_guides/evaluating/usage_pattern.html

Hi, is there a way to do evaluation on RAG agents instead of on the query engine level? I don't see any documentation on that yet

4 comments

LLogan M

Technically, all these methods would work fine for agents. But we don't have anything specific yet for agents (I think in general the agent-space is still figuring out best practices/approaches for this)

GGoat

Do you happen to know if runner.aevaluate_queries takes in langchain agent as input then?

Plain Text

from llama_index.evaluation import BatchEvalRunner

runner = BatchEvalRunner(
    {"faithfulness": faithfulness_evaluator, "relevancy": relevancy_evaluator},
    workers=8,
)

eval_results = await runner.aevaluate_queries(
    vector_index.as_query_engine(), queries=questions
)

LLogan M

ohhh, it wouldn't take a langchain agent. You could make it work by wrapping the langchain agent with a custom query engine though

https://docs.llamaindex.ai/en/stable/examples/query_engine/custom_query_engine.html#defining-a-custom-query-engine

GGoat

i see, thanks

Add a reply