Handling Context Length Exceeding Error in Subquestion Routine

Question

Hey @Logan M how are you? I am trying to use the workflow example to generate an example sub questions, then go through and use react to answer the subquestions form here: https://docs.llamaindex.ai/en/stable/examples/workflow/sub_question_query_engine/

The issue is when I get to the point in the subquestion routine:

agent = ReActAgent.from_tools(
await ctx.get("tools"), llm=llm_4o_2, verbose=False, max_iterations=5
)
response = agent.chat(ev.question)

There are some subquestion quries where it fails with :

Error code: 400 - {'error': {'message': "This model's maximum context length is 128000 tokens. However, your messages resulted in 129643 tokens. Please reduce the length of the messages.", 'type': 'invalid_request_error', 'param': 'messages', 'code': 'context_length_exceeded'}}

I really do not understand how to control this. BTW, the tools is a lit of retriever tools, that was supposed to have node_postprocessor reranker to titrate down the nodes. but i do keep hitting this error regardless.

cmosguy · Answer

yes it is i guess.... maybe i am doing something stupid, but here is what I basically do:

Plain Text

 vector_query_engine = vector_indexes[ns_section].as_query_engine(
                llm=llm_4o_mini,
                similarity_top_k=20,
                node_postprocessors=node_postprocessors,
            )
            summary_query_engine = summary_indexes[ns_section].as_query_engine(
                llm=llm_4o_mini,
            )

 retriever_tools: List[RetrieverTool] = [
                RetrieverTool.from_defaults(
                    retriever=vector_indexes[ns_section].as_retriever(
                        similarity_top_k=20, node_postprocessors=node_postprocessors
                    ),
                    description=vector_tool_description,
                ),
                RetrieverTool.from_defaults(
                    retriever=summary_indexes[ns_section].as_retriever(),
                    description=summary_tool_description,
                ),
            ]

            all_retriever_tools.extend(retriever_tools)

what I do is take that all_retriver_tools object and use that for the tools into the react agent.

cmosguy · Answer

the node_postprocessor is the reranker

cmosguy · Answer

@Logan M thoughts 👆?

Logan M · Answer

node postprocessors do not go into as_retriever() Should be RetrieverTool.from_defaults(retriever=..., node_postprocessors=node_postprocessors)

cmosguy · Answer

oh crap, really?

cmosguy · Answer

thanks so much for catching that!

Find answers from the community

Handling Context Length Exceeding Error in Subquestion Routine