Handling Context Length Exceeding Error in Subquestion ...

At a glance

The community member is trying to use a workflow example to generate sub-questions and then use React to answer them. However, they are encountering an error where the context length exceeds the maximum allowed by the model. The community member is unsure how to control this issue.

In the comments, another community member suggests that the node_postprocessors should not be passed to the as_retriever() method, but rather to the RetrieverTool.from_defaults() method. This seems to be the answer to the issue.

Useful resources

ccmosguy

Hey @Logan M how are you? I am trying to use the workflow example to generate an example sub questions, then go through and use react to answer the subquestions form here: https://docs.llamaindex.ai/en/stable/examples/workflow/sub_question_query_engine/

The issue is when I get to the point in the subquestion routine:

agent = ReActAgent.from_tools(
await ctx.get("tools"), llm=llm_4o_2, verbose=False, max_iterations=5
)
response = agent.chat(ev.question)

There are some subquestion quries where it fails with :

Error code: 400 - {'error': {'message': "This model's maximum context length is 128000 tokens. However, your messages resulted in 129643 tokens. Please reduce the length of the messages.", 'type': 'invalid_request_error', 'param': 'messages', 'code': 'context_length_exceeded'}}

I really do not understand how to control this. BTW, the tools is a lit of retriever tools, that was supposed to have node_postprocessor reranker to titrate down the nodes. but i do keep hitting this error regardless.

6 comments

ccmosguy

yes it is i guess.... maybe i am doing something stupid, but here is what I basically do:

Plain Text

 vector_query_engine = vector_indexes[ns_section].as_query_engine(
                llm=llm_4o_mini,
                similarity_top_k=20,
                node_postprocessors=node_postprocessors,
            )
            summary_query_engine = summary_indexes[ns_section].as_query_engine(
                llm=llm_4o_mini,
            )

 retriever_tools: List[RetrieverTool] = [
                RetrieverTool.from_defaults(
                    retriever=vector_indexes[ns_section].as_retriever(
                        similarity_top_k=20, node_postprocessors=node_postprocessors
                    ),
                    description=vector_tool_description,
                ),
                RetrieverTool.from_defaults(
                    retriever=summary_indexes[ns_section].as_retriever(),
                    description=summary_tool_description,
                ),
            ]

            all_retriever_tools.extend(retriever_tools)

what I do is take that all_retriver_tools object and use that for the tools into the react agent.

ccmosguy

the node_postprocessor is the reranker

ccmosguy

@Logan M thoughts 👆?

LLogan M

node postprocessors do not go into as_retriever()

Should be RetrieverTool.from_defaults(retriever=..., node_postprocessors=node_postprocessors)

ccmosguy

oh crap, really?

ccmosguy

thanks so much for catching that!

Add a reply

Find answers from the community

Handling Context Length Exceeding Error in Subquestion Routine