Find answers from the community

Updated 4 months ago

Hey team, a question about Phoenix (

At a glance

Hey team, a question about Phoenix (arize) + OpenAIAssistantAgent + VectaraAutoRetriever. Currently in this setup it does not seem like Phoenix is collecting any internal steps that are executed by the VectaraAutoRetriever, so far we see only agent steps being collected which contain only the initial question and the final synthesized answer.

Do you have any tips on collecting intermediate steps by the VectaraAutoRetriever as well that would not require wrapping / rewriting of the retriever?

9 comments

LLogan M

Hmm, I don't think the callback manager is being propagated to the components properly

yyasik

Using OpenInferenceTraceCallbackHandler which is then passed to CallbackManager and every context, retriever and query engine down the stack.

I am not seeing however the callback manager being the first class parameter of the vectara auto retriever's ctor

LLogan M

hmm, no idea.

You did retriever = VectaraAutoRetriever(..., callback_manager=callback_manager) ?

Or you set the global handler maybe?

yyasik

no, I am explicitly setting callbacks, not global

LLogan M

I don't have access to vectara to test, but from looking at the code, the above should give insights into retrieval traces and the llm call to build the vector store query

LLogan M

maybe a bug with arize? If you try the llama debug handler with print_trace_on_end=True, you might see the proper trace printed?

yyasik

ok, let me try it, thanks!

yyasik

yeah, I am getting full traces with the llama handler. I guess this is an issue with arize at this point

yyasik

thanks for the suggestion

Add a reply