Hi, after upgrading LlamaIndex and

At a glance

The community member is experiencing an issue where ChatGPT, after upgrading LlamaIndex and moving from LangChain ChatOpenAI to LlamaIndex OpenAI, is no longer able to understand simple sentences like "Hi". Instead, it responds with "I'm sorry but I don't have the information on this question. I will pass the problem to my human colleagues who will be able to assist you further." The community member has provided the code they are using, and has found that LlamaIndex is passing only one ChatMessage, system, even though the full chat history was provided.

In the comments, another community member suggests that there may have been changes on the OpenAI side, and that the issue could be related to the way tokens are counted. They have figured out that the token limit passed in the chat history should be the sum of two limits, not just the history limit as before.

SSeaCat

Hi, after upgrading LlamaIndex and moving from LangChain ChatOpenAI to LlamaIndex OpenAI, suddenly ChatGPT stopped understanding simple sentenses like "Hi". Instead of responsing with greeting, it doesn't know what to say: "I'm sorry but I don't have the information on this question. I will pass the problem to my human colleagues who will be able to assist you further." The model is gpt-3.5 LlamaIndex 0.8.65
Here is the code:

Plain Text

llm_engine = OpenAI(temperature=0, model_name=model_name, openai_api_key=openai_key, max_tokens=response_limit)
llm_predictor = LLMPredictor(llm=llm_engine)
service_context = ServiceContext.from_defaults(llm_predictor=llm_predictor, chunk_size=chunk_size, 
                                                        callback_manager=callback_manager)
vector_store = storage_service.get_vector_store(collection_name, project.db_name)
index = VectorStoreIndex.from_vector_store(vector_store, service_context)
chatmemory = ChatMemoryBuffer.from_defaults(token_limit=history_limit)
query_engine = index.as_chat_engine(verbose=True,chat_mode="context",
                memory=chatmemory,
                similarity_top_k=similarity_top_k,
                system_prompt=prepared_system_prompt, 
                        node_postprocessors=[CustomPostprocessor(context_limit, query_text + prepared_system_prompt)])
response = query_engine.chat(query_text, chat_history=chat_history)

openai.log = 'debug' is not working for some reason, so I even can't see wha't is bein passed to the OpenAI

Update. I found that, for some reason, LlamaIndex is passing only one ChatMessage, system even though I provided the query and full chat history.

2 comments

TTeemu

Yeah I think there were some changes on OpenAI's side. Some things are different and the debug doesn't work for me either, probably need to take a closer look

SSeaCat

I've figured it out. It turned out they changed the algorithm of counting tokens, now they put context into chat history, so the limit passed there should be sum of 2 limits, not just history as before.

Add a reply

Find answers from the community

Hi, after upgrading LlamaIndex and