how to use QueryFusionRetriever with CondensePlusContex...

how to use QueryFusionRetriever with CondensePlusContextChatEngine with use_async=True

Plain Text

def get_chat_engine() -> "CondensePlusContextChatEngine":

    Settings.llm = OpenAI(model="gpt-4o", temperature=0.1)
    
    index =  VectorStoreIndex.from_vector_store(vector_store=vector_store)
    retriever = index.as_retriever(similarity_top_k=3)

    retriever = QueryFusionRetriever(
        [retriever],
        similarity_top_k=4,
        num_queries=4,
        mode="reciprocal_rerank",
        use_async=True,
        verbose=True,
        query_gen_prompt=BOT_QUERY_GEN_PROMPT
    )
    chat_engine = CondensePlusContextChatEngine.from_defaults(retriever=retriever, system_prompt=SUPPORT_BOT_SYSTEM_PROMPT, streaming=True)
    return chat_engine

async def chat(request: ChatRequestBody):
    try:
       
        engine = get_chat_engine()
        response_stream = engine.stream_chat(message, chat_history=history)
        return StreamingResponse(
            stream_generator(response_stream, request.history, request.timezone),
            media_type="application/x-ndjson",
        )

    except Exception as e:
        traceback.print_exc()
        raise HTTPException(
            status_code=500, detail=f"An error occurred while processing the request. {str(e)}"
        ) from e

this the error

RuntimeError: Nested async detected. Use async functions where possible (aquery, aretrieve, arun, etc.). Otherwise, use import nest_asyncio; nest_asyncio.apply() to enable nested async or use in a jupyter notebook.

6 comments

kkapa.ai

Please stand by @payload as I search the knowledge base. I normally take about 20 seconds or so. :wave:

kkapa.ai

----------------------
The error message you're seeing is due to the fact that you're trying to run asynchronous code in a context that doesn't support it. This is common when you're using asynchronous features of LlamaIndex, such as the QueryFusionRetriever with use_async=True, inside a synchronous context.

To fix this, you can use the nest_asyncio library, which allows for asynchronous tasks to be nested. You can apply it at the beginning of your script or notebook. Here's how you can do it:

Plain Text

import nest_asyncio
nest_asyncio.apply()

After applying nest_asyncio, your code should look like this:

Plain Text

import nest_asyncio
nest_asyncio.apply()

def get_chat_engine() -> "CondensePlusContextChatEngine":

    Settings.llm = OpenAI(model="gpt-4o", temperature=0.1)
    
    index =  VectorStoreIndex.from_vector_store(vector_store=vector_store)
    retriever = index.as_retriever(similarity_top_k=3)

    retriever = QueryFusionRetriever(
        [retriever],
        similarity_top_k=4,
        num_queries=4,
        mode="reciprocal_rerank",
        use_async=True,
        verbose=True,
        query_gen_prompt=BOT_QUERY_GEN_PROMPT
    )
    chat_engine = CondensePlusContextChatEngine.from_defaults(retriever=retriever, system_prompt=SUPPORT_BOT_SYSTEM_PROMPT, streaming=True)
    return chat_engine

async def chat(request: ChatRequestBody):
    try:
       
        engine = get_chat_engine()
        response_stream = engine.stream_chat(message, chat_history=history)
        return StreamingResponse(
            stream_generator(response_stream, request.history, request.timezone),
            media_type="application/x-ndjson",
        )

    except Exception as e:
        traceback.print_exc()
        raise HTTPException(
            status_code=500, detail=f"An error occurred while processing the request. {str(e)}"
        ) from e

kkapa.ai

This should resolve the RuntimeError: Nested async detected error you're encountering.

Please note that the code you provided is not directly extracted from the knowledge sources provided, but the solution to the error is based on the information from the knowledge source ().
----------------------
Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:
----------------------

LLogan M

As the error states, you should use nest_asycio.apply or use asream_chat ?

ppayload

Nest asyncio was not working with fast api will try astream chat

LLogan M

For fast api, you can set the loop type to be asyncio

uvicorn.run(..., loop="asyncio")

Add a reply

Find answers from the community

how to use QueryFusionRetriever with CondensePlusContextChatEngine with use_async=True``