Find answers from the community

Updated 4 months ago

I’m noticing that switching from chat

I’m noticing that switching from chat engine to agent adds a ton of latency. Has anyone else experienced this?
L
3 comments
agents generally have more overhead yea
Pretty expected. Agents need to
  • read the chat history, call the llm to write a response or tool to use
  • call the selected tools if any
  • call the llm to interpret the tool response, and either write a final response or another tool call
  • if theres another tool call, repeat as needed
You could write your own agent if can think of a faster way to process. I did have an example of a from-scratch agent using workflows that would be easy to customize
React agent and function calling agents from scratch with workflows here
https://docs.llamaindex.ai/en/stable/module_guides/workflow/?h=workflows#examples
Add a reply
Sign up and join the conversation on Discord