Find answers from the community

Updated 2 months ago

I built a graph on top of an index, I want to use a streaming response, but it doesn't see

I built a graph on top of an index, I want to use a streaming response, but it doesn't seem to be there, what do I do
k
3 comments
Please stand by @ChuanYue as I search the knowledge base. We are experiencing especially high traffic right now and might take longer than usual. :wave:
----------------------
To use a streaming response with LlamaIndex, you need to ensure that both your LLM and your query engine are configured to support streaming.

  1. Configure an LLM that supports streaming: You need to use an LLM that supports streaming and set streaming=True. Currently, streaming is supported by OpenAI and HuggingFace LLMs. Here is an example of how to set up an LLM with streaming:
Plain Text
llm_predictor = LLMPredictor(
    llm=ChatOpenAI(temperature=0, model_name="gpt-3.5-turbo", streaming=True)
)
service_context = ServiceContext.from_defaults(
    llm_predictor=llm_predictor
)

Source ()

  1. Configure your query engine to use streaming: If you are using the high-level API, set streaming=True when building a query engine. Here is an example:
Plain Text
query_engine = index.as_query_engine(
    streaming=True,
    similarity_top_k=1
)

Source ()

If you are using the low-level API to compose the query engine, pass streaming=True when constructing the ResponseSynthesizer:

Plain Text
synth = ResponseSynthesizer.from_args(streaming=True, ...)
query_engine = RetrieverQueryEngine(response_synthesizer=synth, ...)

Source ()

After setting up the streaming, you can use the query method of your query engine to get a streaming response:

Plain Text
response_stream = query_engine.query("Your query here")

And then print the streaming response:

Plain Text
response_stream.print_response_stream()

Source ()
Please ensure that your LLM and query engine are correctly configured for streaming. If you still encounter issues, it might be due to the specific implementation of your graph or index. In that case, you might need to check the specific documentation or examples related to your graph or index.
----------------------
Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:
----------------------
Add a reply
Sign up and join the conversation on Discord