Optimizing Chat Engine Response Times

Question

What are the arguments chat method of chat engine takes and how can I make it faster to give me response@WhiteFang_Jr @Logan M

WhiteFang_Jr · Answer

for querying, the prime requirement is the query along with this you can also pass the chat history if you want

WhiteFang_Jr · Answer

If you are using open source llm then the repsonse time totally depends on your hardware

WhiteFang_Jr · Answer

You can also try streaming the response that way you dont have to wait for the entire response to generate

Pragyan Mohapatra · Answer

How to stream the response from the chat engine

Logan M · Answer

resp = chat_engine.stream_chat(...)
for r in resp.response_gen: print(r, end="", flush=True)Or asyncresp = await chat_engine.astream_chat(...)
async for r in resp.async_response_gen(): print(r, end="", flush=True)

WhiteFang_Jr · Answer

For more you can refer to the docs here as well: https://docs.llamaindex.ai/en/stable/examples/chat_engine/chat_engine_condense_plus_context/#streaming-support

Pragyan Mohapatra · Answer

How do I display this in chat bot , print statements cannot be displayed, right?@WhiteFang_Jr

WhiteFang_Jr · Answer

Here is an example for FastAPI backend: https://discord.com/channels/1059199217496772688/1217020898180075621/1217520651691360277You can refer to this one

Pragyan Mohapatra · Answer

Thanks @WhiteFang_Jr But this doesn’t seem to have custom prompt to be fed. Can we give our custom prompt to this?

WhiteFang_Jr · Answer

Yes, just need to change the default prompt: https://docs.llamaindex.ai/en/stable/module_guides/models/prompts/usage_pattern/#accessing-prompts

Find answers from the community

Optimizing Chat Engine Response Times