Find answers from the community

Updated 2 months ago

I got this error

I got this error:
Error: This model's maximum context length is 4097 tokens. However, your messages resulted in 4318 tokens (2877 in the messages, 1441 in the functions). Please reduce the length of the messages or functions.
And my memory is configured like this:
_memory = ChatMemoryBuffer.from_defaults( token_limit=2000, chat_history=chat_history )
L
e
14 comments
Try decreasing the token limit?
yess, with 1000 works. But why my messages exeed the model with 2877 tokens if the limit is 2000?
Likely some token counting errors πŸ˜… What chat engine are you using? Agent? Context?
I am using OpenAIAgent. This is my constructor:
self._user_id = user_id self.project_id = index_dir self._tools = {tool.metadata.name: tool for tool in tools} _vector_store = FaissVectorStore.from_persist_dir( f"./llama_index/{index_dir}" ) _storage_context = StorageContext.from_defaults( vector_store=_vector_store, persist_dir=f"./llama_index/{index_dir}", ) _index = load_index_from_storage(storage_context=_storage_context) _memory = ChatMemoryBuffer.from_defaults( token_limit=1000, chat_history=chat_history ) similarity_top_k = 7 _retriever = VectorIndexRetriever( index=_index, similarity_top_k=similarity_top_k ) _query_engine = RetrieverQueryEngine(retriever=_retriever) query_engine_tool = QueryEngineTool.from_defaults( query_engine=_query_engine, ) _all_tools = [query_engine_tool] for tool in tools: _all_tools.append(tool) self.token_counter = TokenCountingHandler( tokenizer=tiktoken.encoding_for_model("gpt-3.5-turbo").encode, ) callback_manager = CallbackManager([self.token_counter]) self._agent = OpenAIAgent.from_tools( _all_tools, llm=llm, callback_manager=callback_manager, memory=_memory, system_prompt=TEXT_QA_SYSTEM_PROMPT.content, ) self._chat_history = chat_history
Any ideas @Logan M ??
I set the limit to 1000 and I am getting the same error
πŸ€·β€β™‚οΈ Not sure man. The EXACT same error? Do you have too many tools?
yess, I have 42
yes the same error: Error: This model's maximum context length is 4097 tokens. However, your messages resulted in 4115 tokens (1949 in the messages, 2166 in the functions). Please reduce the length of the messages or functions.
so on one hand, the chat memory buffer is maybe not limiting tokens properly for some reason. So that needs debugging, but not sure when I can look into that right now

But also, 42 tools is kind of insane. I would use gpt-4 or gpt-3.5-turbo-16k if you really need that many tools. You could also look into a tool retriever agent
https://gpt-index.readthedocs.io/en/stable/examples/agent/openai_agent_retrieval.html#retrieval-augmented-openai-agent
okay thanks, using Retrieval-Augmented OpenAI Agent I reduce the tokens in the functions??
yup, because you only retrieve a subset of functions that are most relevant to the query
rather than sending them all
okass thank youu
Add a reply
Sign up and join the conversation on Discord