Find answers from the community

Updated 4 months ago

Hi.

Hi.
I am using a react agent with step-wise execution. It works well but i cannot find a way to stream the outcome of each step.
I went through the documentation several times in vain.
Any idea?
Thank you
L
N
19 comments
isn't it just .stream_step() ?
I tried but it did not return a generator.
Will double check in case I missed something.
Thanks
pretty sure it does, but happy to take a look at your code if you are stuck
Thanks for your help.
My code is as follows:

Plain Text
from llama_index.core.agent import AgentRunner, ReActAgentWorker, ReActAgent
prompt="An interesting question?"
task = agent.create_task(prompt)
step_output = agent.stream_step(task.task_id)
for token in step_output.output.response_gen:
    print(token, end="")

I get:
ValueError: response_gen is only available for streaming responses. Set is_dummy_stream=True if you still want a generator.

When there is no step left, I do get a generator.
I have been looking into again.
The thing I don't get is: how do I get a generator when I call stream_step? It seems to be somewhere in the output but I cannot iterate over it. Also, the is_dummy_stream seems to only stream the response but after the generation is completed.
Am I wrong? Thanks for your help
This works for me πŸ‘€

Plain Text
llm = OpenAI(model="gpt-3.5-turbo")
agent = OpenAIAgent.from_tools(
    [multiply_tool, add_tool], llm=llm, verbose=True
)

task = agent.create_task("What is 2123 * 215123")

while True:
    task_result = agent.stream_step(task.task_id)
    for token in task_result.output.response_gen:
        print(token, end="")
        time.sleep(0.2) # sleeping, just to emphasize the streaming
    
    if task_result.is_last:
        break
Not every step will have anything to stream
Since for example, the first step is just calling the multiply tool
Thanks.
Tried it and got:
Plain Text
Cell In[18], line 4
      2 while True:
      3     task_result = agent.stream_step(task.task_id)
----> 4     for token in task_result.output.response_gen:
      5         print(token, end="")
      6         time.sleep(0.2) # sleeping, just to emphasize the streaming

File ~/Desktop/project/venv/lib/python3.10/site-packages/llama_index/core/chat_engine/types.py:73, in AgentChatResponse.response_gen(self)
     71 """Used for fake streaming, i.e. with tool outputs."""
     72 if not self.is_dummy_stream:
---> 73     raise ValueError(
     74         "response_gen is only available for streaming responses. "
     75         "Set is_dummy_stream=True if you still want a generator."
     76     )
     78 for token in self.response.split(" "):
     79     yield token + " "

ValueError: response_gen is only available for streaming responses. Set is_dummy_stream=True if you still want a generator.


Upgraded to llama-index 0.11 just in case but still the same problem
You copied the code above?
Yes but i am not using chatgpt
I am using an open source llm
And my agent has 2 query engine tools
Oh, so you are using the FunctionCallingAgent then? It doesn't support streaming right now
Hoping to change that sometime in the next week or two
I create the agent like this:
agent = ReActAgent.from_tools(
query_engine_tools,
llm=Settings.llm,
verbose=True,
memory=memory,
max_iterations=15
)
And the query_engine_tools are created like so:

query_engine_tools = [
QueryEngineTool(
query_engine=query_engine_n,
metadata=ToolMetadata(
name="Nat",
description="Useful for XYZ",
),
),
QueryEngineTool(
query_engine=query_engine_d,
metadata=ToolMetadata(
name="Dir",
description="Useful for ZYX",
),
),
]
Not sure this is what I am doing here
That would be great!
Thanks for your work πŸ™‚ πŸ™Œ
Add a reply
Sign up and join the conversation on Discord