While in the second case where I am

While in the second case where I am trying to use QueryEngineRouter, it fails. Below is the code for the same.

Plain Text

# construct list_index and vector_index from storage_context and service_context
list_index = ListIndex(nodes, service_context=service_context)
vector_index = VectorStoreIndex(nodes, service_context=service_context)

# define list_query_engine and vector_query_engine
list_query_engine = list_index.as_query_engine(
    response_mode="tree_summarize",
    use_async=True,
)
vector_query_engine = vector_index.as_query_engine()

list_tool = QueryEngineTool.from_defaults(
    query_engine=list_query_engine,
    description="Useful for summarization questions related to the data source",
)
vector_tool = QueryEngineTool.from_defaults(
    query_engine=vector_query_engine,
    description="Useful for retrieving specific context related to the data source",
)

# construct RouterQueryEngine
query_engine = RouterQueryEngine(
    # selector=LLMSingleSelector.from_defaults(),
    selector=PydanticSingleSelector.from_defaults(),
    query_engine_tools=[
        list_tool,
        vector_tool,
    ],
)
response_str = query_engine.query(
    "What is the maximum quantity that can be submitted for a First Leg Order in trading?"
).response

And the http log gives 404 Not Found.

Plain Text

HTTP Request: POST https://visdam-labs.openai.azure.com/chat/completions "HTTP/1.1 404 Resource Not Found"

Please let me know if I am doing something wrong here. Thank you for looking into it.

17 comments

bbmax

your service context is the same for both?

LLogan M

Plain Text

query_engine = RouterQueryEngine(
    # selector=LLMSingleSelector.from_defaults(),
    selector=PydanticSingleSelector.from_defaults(llm=service_context.llm),
    query_engine_tools=[
        list_tool,
        vector_tool,
    ],
    service_context=service_context
)

LLogan M

should probably help

ddisco.dr

For the first (successful case)

Plain Text

# Initialize Service Context
service_context = ServiceContext.from_defaults(
    llm=llm,
    embed_model=embed_model,
    chunk_size=512,
    chunk_overlap=20,
    callback_manager=callback_manager,
)
set_global_service_context(service_context)

While for the second (failure case):

Plain Text

# Initialize Service Context
service_context = ServiceContext.from_defaults(
    llm=llm,
    embed_model=embed_model,
    chunk_size=512,
    chunk_overlap=20,
)
set_global_service_context(service_context)

bbmax

sorry I guess I should ask more specifically the llm

LLogan M

Yea, since the selector needs an LLM as a kwarg, it's not picking it up from the global service context you set

LLogan M

(Super annoying I know, v0.10 is going to remove the service context I think LOL)

ddisco.dr

Yes! This helped. 👏🏻 Shouldn't by default PydanticSingleSelector.from_defaults() method pick the llm from the global service_context?

LLogan M

its a long story, but no 🙂

LLogan M

Once we remove the service context, it should make a lot more sense

LLogan M

its a weird limbo state at the moment

ddisco.dr

Understood. And thanks again. Would be great if you can share a document if we have any.

LLogan M

I don't think I have a document on this -- but just looking at kwargs, the pydantic selector doesn't take in a service context (and therefore, implies it doesn't interact with it), just an LLM as a kwarg

If you ever see llm as a kwarg to an interface, you can assume it does not interact with the service context

ddisco.dr

No worries. Thank you so much for the explanation.

ddisco.dr

Just to add – If I use LLMSingleSelector, I don't need to pass the LLM object in the function. It works perfectly fine.

LLogan M

That makes sense

Attachment

LLogan M

🫠

Add a reply

Find answers from the community

While in the second case where I am