oof, it's off my like 5 tokens too
probably a small token counting error :PSadge:
In the service context, try slightly lowering the context window -- this will help account for minor errors in token counting
(openai does not make token counting easy lol)
ok, so it comes from token counting
service_context = ServiceContext.from_defaults(..., context_window=16000)
also, for the index retriever, what is the size of each document from the top_k ?
The retriever is fetching nodes, and the size of nodes is determined by the chunk_size (default is 1024 tokens)
ok so chunk size is per default 1024
so no problem to put top k at 5 instead of 3
it seems to work now by lowering the context size
also, is there a way to make the engine async (especially the summary engine)
(and, when asking to resume each document, when there is like 15 documents, it refuses, but I can understand if it's normal lol)
yea -- response = await index.aquery("query str")
ow ok thanks, it's possible, I've seen through the doc but didn't search for all the possibilities with async
when using directly the query engines to a router query engine, then to condense question chat engine, do you know how to proceed ?
response = await chat_engine.achat("msg") should work
Ok, it will not apply to the execution of the "ramake" question asked to the llm, but also to the query engines ?
it does not seem to work in my case: File "/tmp/8dbef6bbe34759f/RoboQOF.py", line 184
await strmlt.session_state.conversation.achat(user_prompt)
^
SyntaxError: 'await' outside async function
You have to put aync in front of the def for that function to indicate it should run asynchronously.
yea async in python is fun
ow ok, sry, I never worked with that before
Ok, so, afer applying this, there is no error, but the chat session state return nothing
Need I put out the second line of the handling function, and call the chat history after the async function ?
and what is the difference with the use_async=True parameter ?
