Find answers from the community

Updated 4 months ago

Hey team, a question about Phoenix (

At a glance
Hey team, a question about Phoenix (arize) + OpenAIAssistantAgent + VectaraAutoRetriever. Currently in this setup it does not seem like Phoenix is collecting any internal steps that are executed by the VectaraAutoRetriever, so far we see only agent steps being collected which contain only the initial question and the final synthesized answer.

Do you have any tips on collecting intermediate steps by the VectaraAutoRetriever as well that would not require wrapping / rewriting of the retriever?
L
y
9 comments
Hmm, I don't think the callback manager is being propagated to the components properly
Using OpenInferenceTraceCallbackHandler which is then passed to CallbackManager and every context, retriever and query engine down the stack.

I am not seeing however the callback manager being the first class parameter of the vectara auto retriever's ctor
hmm, no idea.

You did retriever = VectaraAutoRetriever(..., callback_manager=callback_manager) ?

Or you set the global handler maybe?
no, I am explicitly setting callbacks, not global
I don't have access to vectara to test, but from looking at the code, the above should give insights into retrieval traces and the llm call to build the vector store query
maybe a bug with arize? If you try the llama debug handler with print_trace_on_end=True, you might see the proper trace printed?
ok, let me try it, thanks!
yeah, I am getting full traces with the llama handler. I guess this is an issue with arize at this point
thanks for the suggestion
Add a reply
Sign up and join the conversation on Discord