Streaming Text Down To The Client

Question

Hello all! I've managed to build an AgentWorkflow with human in the loop support. The workflow works great, the only thing I'm not able to achieve is streaming text down to the client. I'm using the suggested approach:async for event in handler.stream_events(): if isinstance(event, AgentStream): message = ChatMessage(role="bot", type="delta", content=event.delta) await websocket.send_json(message.model_dump())
...The message sent to the client always have an empty delta. I'm using OpenAI with streaming set to true, is there anything I'm missing?

Logan M · Answer

This is slightly annoying, openai doesn't have delta's for tool calls, and calculating one yourself is very tricky haha but it should get you somewhere

Ariel · Answer

I see! I'm inspecting tool_calls as you suggested. That explains why, the text I want to stream is the output of a tool call, so there's not support for this currently 😥
Hopefully, they will add it at some point. Thanks!

Logan M · Answer

I tried it add it and sunk many errors, but since OpenAI is streaming it as a completed JSON and updating it in place, streaming it in the same delta fashion is almost impossible. I don't see it being added anytime soon tbh (if its even possible)

I would instead just stream the latest version of the tool call, and use that to update some text in a streamlit or frontend ?

Ariel · Answer

Do you mean to somehow simulate the streaming effect on the client ? Not sure it's worth the effort... have you ever done that?

Logan M · Answer

If you are streaming to a react component, its pretty easy to do -- just updating the text in-place

Logan M · Answer

if its purely over an API, yes, very hard haha

Ariel · Answer

so basically a react component that grabs the final output text received and renders it by chunks to fake the streaming effect... 🤔

Ariel · Answer

I'll ask ChatGpt about it, lol

Logan M · Answer

I think it's as simple as just refreshing the text being displayed Pseudo code beingcomponent.text = "Hello"
component.text = "Hello world"

Logan M · Answer

Theres definitely a react/js way to do that while a response/events strean from an api

Find answers from the community

Streaming Text Down To The Client