Find answers from the community

Updated 11 months ago

Chat

Chat Engine is stateful: https://docs.llamaindex.ai/en/stable/module_guides/deploying/chat_engines/root.html

What is the idiomatic way to implement it in a setup where the code runs on multiple servers, possibly behind a load balancer? In this setup, two consecutive chat questions may not reach the same server, so question 2 won't have the state from question 1 and answer 1.
L
S
3 comments
You can avoid the state by passing in the chat history

chat("hello", chat_history=chat_history)
Chat history is a list of ChatMessage objects
Wow, that's straightforward! Thanks!
Add a reply
Sign up and join the conversation on Discord