Hi guys, in an agent `.chat()` response

CChicoButico

Hi guys, in an agent .chat() response, I'm trying to collect information about the steps executed by the agent. I mainly need to know the source of that response. Most of the time, I'm using high level APIs for simplicity, but if you think I have some way making this customization going to a low-level api, I'll be glad to hear from you.

When my agent calls a RetrieverQueryEngine, the response contains a source_nodes prop and this is great. I'm able to identify the nodes used to compose that response.

But when this agent does a call to another agent (let's say a sub-agent: I'm using it for function calling with ObjectIndex), I only get source for the first tool, nothing that the subagent calls is accessible in the final response.

I want to obtain the retrieved nodes from ObjectIndex and identify what function was called. I'll post an image in the thread for clarification.

6 comments

CChicoButico

Attachment

CChicoButico

I did some debugging, and I feel we're missing the opportunity to store the source_nodes here: https://github.com/run-llama/llama_index/blob/be63bae53227f1360472477eb2afa993791c09ce/llama-index-core/llama_index/core/objects/base.py#L47-L49

but I don't know how we could return that in that function because it'll probably change the method/interface signature

vverdverm

I'm just getting to agent stuff, but I've been going lower level (and avoiding the high-level) apis for this exact reason(s)

Need to customize better
Need to bring more of the inner details to the surface

For example, I want to be able to highlight the parts of the documents used in supplying the answer in basic RAG chat (i.e. what was returned from the vector db and was added to the system prompt)

CChicoButico

Yeah, I also need to do that and it works great for the RetrieverQueryEngine as it populates the source_nodes and I'm able to access it in the response. But ObjectRetriever is not saving the source_nodes to the response. So even though the nodes are object functions metadata, I'd like to have access to those.

LLogan M

You are definitely right, we should be storing source nodes here. (If you opened a PR, I would definitely review it ❤️ )

CChicoButico

Cool, I'll take some time to try to fix it. Thanks, Logan!

Add a reply

Find answers from the community

Hi guys, in an agent `.chat()` response