Hi guys, in an agent `.chat()` response

At a glance

The community member is trying to collect information about the steps executed by an agent in an agent.chat() response, specifically the source of the response. They are using high-level APIs for simplicity, but are open to using low-level APIs if necessary.

When the agent calls a RetrieverQueryEngine, the response contains a source_nodes property, which the community member can use to identify the nodes used to compose the response. However, when the agent calls another agent (a sub-agent), the community member only gets the source for the first tool, and not the sub-agent calls.

The community member wants to obtain the retrieved nodes from ObjectIndex and identify what function was called. Some community members have suggested that the source_nodes should be stored in a specific location in the code, but they are unsure how to do this without changing the method/interface signature.

There is no explicitly marked answer in the comments.

Useful resources

CChicoButico

Hi guys, in an agent .chat() response, I'm trying to collect information about the steps executed by the agent. I mainly need to know the source of that response. Most of the time, I'm using high level APIs for simplicity, but if you think I have some way making this customization going to a low-level api, I'll be glad to hear from you.

When my agent calls a RetrieverQueryEngine, the response contains a source_nodes prop and this is great. I'm able to identify the nodes used to compose that response.

But when this agent does a call to another agent (let's say a sub-agent: I'm using it for function calling with ObjectIndex), I only get source for the first tool, nothing that the subagent calls is accessible in the final response.

I want to obtain the retrieved nodes from ObjectIndex and identify what function was called. I'll post an image in the thread for clarification.

6 comments

CChicoButico

Attachment

CChicoButico

I did some debugging, and I feel we're missing the opportunity to store the source_nodes here: https://github.com/run-llama/llama_index/blob/be63bae53227f1360472477eb2afa993791c09ce/llama-index-core/llama_index/core/objects/base.py#L47-L49

but I don't know how we could return that in that function because it'll probably change the method/interface signature

vverdverm

I'm just getting to agent stuff, but I've been going lower level (and avoiding the high-level) apis for this exact reason(s)

Need to customize better
Need to bring more of the inner details to the surface

For example, I want to be able to highlight the parts of the documents used in supplying the answer in basic RAG chat (i.e. what was returned from the vector db and was added to the system prompt)

CChicoButico

Yeah, I also need to do that and it works great for the RetrieverQueryEngine as it populates the source_nodes and I'm able to access it in the response. But ObjectRetriever is not saving the source_nodes to the response. So even though the nodes are object functions metadata, I'd like to have access to those.

LLogan M

You are definitely right, we should be storing source nodes here. (If you opened a PR, I would definitely review it ❤️ )

CChicoButico

Cool, I'll take some time to try to fix it. Thanks, Logan!

Add a reply

Find answers from the community

Hi guys, in an agent `.chat()` response