Find answers from the community

Updated 4 months ago

When streaming a chat with a react agent

At a glance

The post describes an issue with streaming a chat using a React agent, where the community member is getting strange results. The code provided shows the setup of a ReActAgent using a QueryEngineTool. The comments suggest that the issue may be related to the use of an LLM, with some community members speculating that it could be Claude 2 and that Anthropic has a tendency to use XML in their outputs. The comments also discuss the challenges of getting multimodal chat working across different systems, and a community member asks for examples of best practices for implementing the stream_chat method in the Hugging Face codebase.

Useful resources
When streaming a chat with a react agent i get some strange results:
Plain Text
def get_react_agent(
    vector_store: PineconeVectorStore,
    history: List[ChatMessage],
    user_id: str,
    model: ChatLLM,
) -> ReActAgent:
    query_engine = get_query_engine(
        user_id=user_id, vector_store=vector_store, model=model
    )
    query_engine_tool = QueryEngineTool.from_defaults(query_engine=query_engine)

    return ReActAgent.from_tools(
        tools=[query_engine_tool],
        llm=model.value,
        chat_history=history,
    )
           
agent = get_react_agent(
        vector_store=get_pinecone_vector_store(),
        history=history,
        user_id=user_id,
        model=model,
    )
# Valid types based on looking at source code
response = agent.stream_chat(message)  # pyright: ignore

... generator websocket push code ...

print(response.response)
#<function_calls>
#<invoke>
#<tool_name>query_engine_tool</tool_name>
#<parameters>
 # "input": "hello"
#</parameters>
#</invoke>
#</function_calls>

#Thought: I need to use the query_engine_tool to help me understand and respond to the user's input.

#Action: query_engine_tool

#Action Input: {"input": "hello"}
L
m
15 comments
lol seems like the LLM tried to write some xml?
Definitely a symptom of the LLM being used
yeah I guess the quality of anthropic has fallen greatly
anthropic really does like XML from my experience
We almost need an XML mode for these kinds of structured outputs
So much work though lol
Yeah that's weird, gonna try and chat with claude without ReAct and see if it's better
Not as easy as it seems to get multimodel chat working across everything
multimodal support is pretty jank at the moment -- hopefully that improves
yeah im gonna have to get gemeni in soon here
@Logan M I am seeing under the hugging face class:
Plain Text
    def stream_chat(
        self, messages: Sequence[ChatMessage], **kwargs: Any
    ) -> ChatResponseGen:
        raise NotImplementedError

I kind of want to implement this, any good examples of where to look for best practices inside the codebase
πŸ‘ ill see what i can do, thanks!
Add a reply
Sign up and join the conversation on Discord