Well it looks like it simply does not

ttilleul

Well it looks like it simply does not work at all in fact ... I don't know why this "haiku" query was streamed ...

Here's a simpler example:

Plain Text

from llama_index.llms import OpenAI
from llama_index import ServiceContext
from llama_index.chat_engine import SimpleChatEngine

import os, openai

os.environ['OPENAI_API_KEY'] = "sk-..."
openai.api_key = os.environ['OPENAI_API_KEY']


service_context = ServiceContext.from_defaults(
    llm=OpenAI(temperature=0.7, model="gpt-3.5-turbo")
)
chat_engine = SimpleChatEngine.from_defaults(service_context=service_context)
response = chat_engine.stream_chat("Why is the sun yellow ?")
# for token in response.response_gen:
    # print(token, end="")
    
response.print_response_stream()

61 comments

LLogan M

Hmm, let me try this locally

LLogan M

Hmm, seems to stream just fine for me

Attachment

ttilleul

ok let me try on another computer ...

ttilleul

well AT FIRST it worked .... but I was on llama_index 0.7.4 (I think) ... upgraded to 0.7.17 and it stopped working (on that new computer) ...

Will try to downgrade ...

ttilleul

yup ... 0.7.4 streaming works ... not 0.7.17 ... trying now 0.7.16 ...

ttilleul

0.7.16 no streaming

ttilleul

0.7.10 crashes with 'StreamingAgentChatResponse' object has no attribute 'print_response_stream'

ttilleul

0.7.14 no streaming

LLogan M

And this is all with the same code above?

ttilleul

0.7.12 no streaming

ttilleul

yes

LLogan M

Mind blowing tbh, the latest version should be working. That's what I used above

LLogan M

How do you know it's no streaming?

ttilleul

0.7.11 same crash as 0.7.10

LLogan M

If the response is short enough, it might appear like it's not streaming

ttilleul

0.7.8 same crash

LLogan M

Could prompt it to write something longer, like a short story

ttilleul

yes, but there's also the "wait" ...

ttilleul

yes you can try with the "write me a poem" query

ttilleul

0.7.8 crash

LLogan M

older versions won't have that attribute

ttilleul

0.7.6 crash

ttilleul

0.7.4 works though ...

ttilleul

0.7.5 works

LLogan M

Right, but the response type changed from StreamingResponse to StreamingAgentResponse, so that we can align the response types for agents/chat engines

LLogan M

I can't reproduce on my end, so I'm really not sure what the issue is 🤔

ttilleul

So to sum up: 0.7.5 was the last working ... from 0.7.6 to 0.7.11 : crash ... 0.7.12 and onwards no crash but no stream ...

LLogan M

Maybe a fresh venv would help

ttilleul

can you try with 0.7.4 ?

ttilleul

I'm not using venv (yes I know ... :p )

bro

plz

hahaha

lol

on bash:

Plain Text

python -m venv venv
source venv/bin/activate
pip install llama-index

ttilleul

nooooo don't make me do this 🙂

I'm making you

😆

arf ! lol ! 🙂

Trust me, it makes python dev so much better

LLogan M

I used to be opposed too, but I've been converted haha

ttilleul

So I was told ... but no problem so far ... and ... that's another thing I need to "tame" ... 🙂

LLogan M

Then you'll know for sure your env is using the right versions of packages 🙂

ttilleul

ok ... I need to tame "venv" before going further ... I'm on Win11, folders are sync'd by dropbox across several work computers etc ... 🙂

ttilleul

ok still not working (I think) ...

ttilleul

let me try something ...

LLogan M

are you using WSL? or powershell?

ttilleul

ok, this works:

Plain Text

for token in response.response_gen:
    print(token, end="", flush=True)

instead of response.print_response_stream()

Notice the flush=True that forces flushing the output stream

LLogan M

oh the flush!

LLogan M

I need to add that!

ttilleul

I'm using "WT" which is the new "windows terminal" ...

LLogan M

d;oh

ttilleul

🙂

LLogan M

actually super important haha. Will merge a PR shortly 🙂

ttilleul

cool ! glad I could help ! 🙂

LLogan M

Thanks for the patience and testing!

ttilleul

no problem ...

ttilleul

really loved your videos on the YT channel, waiting for more ...

ttilleul

also I might have a couple of other questions in the next few days 😉

LLogan M

Hey thanks! I'm just making videos as I have time to actually build the project out. Glad you like them!

LLogan M

And no worries, happy to help!

Add a reply

Find answers from the community

Well it looks like it simply does not