Find answers from the community

Updated 9 months ago

hey guys, hope everything going great!

hey guys, hope everything going great!
just a quick question, in my app I would like to stream model response but also return retrieved context nodes, which I believe could be returned the moment it starts generating the response. Is there any standard way do so? Thanks a lot!

4 comments

TTeemu

You can do

Plain Text

query_engine = index.as_query_engine(streaming=True, similarity_top_k=1)

streaming_response = query_engine.query(
    "What did the author do growing up?",
)

Then to print it

Plain Text

streaming_response.print_response_stream()

For the response source nodes you can do

Plain Text

print(response.source_nodes)

TTeemu

https://docs.llamaindex.ai/en/stable/module_guides/deploying/query_engine/streaming/

TTeemu

This will print the source nodes first and then start streaming the response

Plain Text

query_engine = index.as_query_engine(streaming=True, similarity_top_k=1)

streaming_response = query_engine.query(
    "Write me a long essay about the context",
)

print(streaming_response.source_nodes)
streaming_response.print_response_stream()

eemmepra

thanks a lot man!

Add a reply