Update defaults by logan-markewich · Pul...

Also for some reason, now the chat engine refuse to answer query with "I'm sorry, but I cannot provide a <query> based on the given context information".

Something that not experienced in 0.7.23

Create an issue for tracking
https://github.com/jerryjliu/llama_index/issues/7231

Can you give more details on the chat engine @Rendy Febry ?

Would love to reproduce locally

Prompts are a tough problem -- just wanted to make our prompts work better with gpt-3.5 and gpt-4, since they were all optimized for gpt-3 originally

@Logan M Not sure if I can give more info, as the result contains our company data.

All I can say is the query is something like this "Please summary <topic/event>".
When queried the retriever successfully pull articles/corpus that related to that topic/event and inject that to context window.
But the chat engine refuse to answer and only give "I'm sorry, but I cannot provide a <bla bla bla> based on the given context information"

I guess this is related to the additional prompt


Interestingly enough, even though OpenAIAgent refuse to answer, the ContextChatEngine still works as expected.

That prompt is only used in the query engine 🤔

Which chat engine gives that response? Just the openai one? Or the actual query engine is giving that response?

I can maybe remove that part of the prompt. Was just an attempt to get the model to hallucinate less (which seemed to work on our test data anyways)

I only tried these two

• OpenAIAgent => Can not answer, it only give "I'm sorry, but I cannot provide a <bla bla bla> based on the given context information"
• ContextChatEngine => Can answer as expected

That interesting though, because naturally agent should be more flexible to answer isn't it?

For example if I query with "Tell me a joke":

• The OpenAIAgent should not use the QueryEngineTool at all, and should give the joke directly from llm own knowledge.
• In the opposite ContextChatEngine should always query the index, and and should not answer if the give context is not related to the query.

But from my observation, the behavior looks inverted.

Well actually, they should both be equally flexible

The context chat engine just retrieves context on every query, and puts it in the system prompt. It can chose to use that information or answer without it by default. A more strict system prompt will change that behaviour

The openai agent really confuses me actually. Let me try locally 🙂

Some prelim checking, but the openai agent works fine for me with "tell me a joke" type queries, as well as queries specific to the tools it has 🤔

The context chat engine just retrieves context on every query, and puts it in the system prompt. It can chose to use that information or answer without it by default. A more strict system prompt will change that behaviour

Yes I understand the core idea of ContextChatEngine, very attractive actually. But I prefer to use OpenAIAgent for a few reasons.


For example the user input is, "Hi My name is Rendy, and know about the Russo-Ukraine War, what do you know about that event?"
ContextChatEngine will use the whole input as input for QueryEngine,
While OpenAIAgent will only use the "Russo-Ukraine War" as input.


So if you have a system that show the source_nodes as UI element, it because doesn't make sense to user eye. Because they don't know if that sources_nodes actually used to generate the response or not.
For OpenAIAgent, if the agent decided to not use the QueryEngineTool, the source_nodes from response object remain empty. So user immediately know.

yea that's fair, was just clarifying how both work 🙂

should we stick with 0.7.23 ? which version we should use from langchain? my program was broke now. need to fix it quickly.

i will modify my requirement and stick with 0.7.23 at current moment. Will the NLTK impacts the embedding file generated before ? we should do a new embedding ?

No need to re-embed, it will only change new data being inserted 👍