Hey

TTikGrig

Hey!

I’ve changed the chat_mode=“condense_question” to chat_mode=“openai” and now I am getting lots of random responses outside the context. For the same question, condense_question mode was able to answer within the context, while openai/best did’t (answered outside the context. Noticed that it doesn’t call function everytime when I send a message.)

Is there a way to fix it? Or is there a better way to implement OpenAIAgent in my case? (Still learning, so sorry I probably lack some knowledge)

6 comments

LLogan M

yea, the openai mode is a little different. It will query the index/tool based on the description of it. But from as_chat_engine the description might be too generic.

You could set it up from scratch in order to write a better description
https://gpt-index.readthedocs.io/en/latest/examples/agent/openai_agent_with_query_engine.html

Another option is a chat engine I just added today that I actually really like. It will retrieve text from the index every time using the user message, and insert that as context in the system prompt.

Then, it can respond using that context OR respond to normal iteractions like "hello" without making more than one LLM call

It's not released on pypi yet, but you could install from source to get it now
pip install git+https://github.com/jerryjliu/llama_index

https://gpt-index.readthedocs.io/en/latest/examples/chat_engine/chat_engine_context.html

TTikGrig

Oh, great, thanks a lot!

Another thing I wanted to share here is this:

The output from tool answers correctly (withing context), however, when I use:

try:
chat_engine = initialize_chat_engine(user_id)
except Exception as e:
logging.error(f"Error occurred while initializing chat engine: {e}")
return

#(enable if it's chat_engine, instead of query_engine)
#chat_engine.reset()
response = chat_engine.chat((f'Use the tool to answer: {text}'))
logging.debug(f"Response: {response}")
if isinstance(response, AgentChatResponse):
message_text = response.response

to show the response, I get response outside of context.

Attachments

LLogan M

Is this with a vector index? Have you adjusted the top k or chunk size at all?

TTikGrig

Yes, it's a vector index.

I did specify chunk size, but I guess it's no use coz I made it maximum (i believe):

prompt_helper = PromptHelper(
context_window=4096,
num_output=1200,
chunk_size_limit=2048,
chunk_overlap_ratio=0.2
)

service_context = ServiceContext.from_defaults(llm=llm,callback_manager=callback_manager, prompt_helper=prompt_helper)

I saw that chunk_size_limit is deprecated, but was getting this error after changing: PromptHelper.init() got an unexpected keyword argument 'chunk_size' and tbh didn't go deep into it yet.

LLogan M

Actually in general, the prompt helper itself is a little deprecated.

It's best to set these values in the service context itself.

Also I think setting the chunk size like that may be causing issues? Try not using the prompt helper, and maybe doing something like this, you may see improved results actually

Plain Text

from llama_index import ServiceContext, set_global_service_context 
from llama_index.llms import OpenAI

llm = OpenAI(model="gpt-3.5-turbo", temperature=0)
service_context = ServiceContext.from_defualts(llm=llm, chunk_size=1024)
set_global_service_context(service_context)

And since I set a global, no need to pass it in anywhere 👌

TTikGrig

Got it, thanks for the help and patience! 😄 Appreciate it!

Add a reply

Find answers from the community

Hey