Chat engine

At a glance

The community members are trying to mix indexed documents with chat history to create a QA chatbot using LlamaIndex. They are facing issues where the chatbot is unable to answer questions about LlamaIndex or other broader topics, as the context is limited to the provided documents. The community members have tried using different language models (GPT-3.5-turbo and GPT-4) and modifying the prompt, but the results are still unsatisfactory. They are exploring ways to combine the knowledge from the documents and the language models to provide more comprehensive answers.

Useful resources

TT. Ventura

Hey Guys,

How do I mix indexed docs with a chat history ?

I'm trying something like this:

Plain Text

...
chat_history = [
    ChatMessage(role=MessageRole.SYSTEM, content="You are a helpful QA chatbot that can answer questions about llama-index."),
    ChatMessage(role=MessageRole.USER, content="How do I create an index?"),
    ChatMessage(role=MessageRole.ASSISTANT, content="LlamaIndex is a data framework for LLM-based applications which benefit from context augmentation."),
]

llm = OpenAI(model="gpt-3.5-turbo", temperature=0)
llm.chat(chat_history) # here I try to attach the chat history

documents = SimpleDirectoryReader("data").load_data() # this is just my CV in text
index = VectorStoreIndex.from_documents(documents)

query_engine = index.as_query_engine()
response = query_engine.query("What is LlamaIndex ?")
print(response)

Response: "LlamaIndex is not mentioned in the provided context information."

20 comments

WWhiteFang_Jr

You'll have to use chat engine for making your query use previous chats to form question and then answer.

WWhiteFang_Jr

https://docs.llamaindex.ai/en/stable/module_guides/deploying/chat_engines/usage_pattern.html#usage-pattern

TT. Ventura

Nice doc. Thanks for that. But the behaviour is a bot weird. Let me try to explain.

TT. Ventura

I'm indexing my CV as a test and the chat History.

TT. Ventura

If I ask the fist time

response = chat_engine.chat("What is LlamaIndex ?")

It gives the right answer.

If I try

response = chat_engine.chat("What is LlamaIndex ? Who's Thiago ?")

"The context does not provide any information about "LlamaIndex". Thiago Ventura is a professional web developer..."

TT. Ventura

if I try again the first question "What is LlamaIndex ?"

TT. Ventura

"The context provided does not contain any information about LlamaIndex or its use in LLM-based applications that benefit from context augmentation."

TT. Ventura

my updated code:

Plain Text

chat_history = [
    ChatMessage(role=MessageRole.SYSTEM, content="You are a helpful QA chatbot that can answer questions about llama-index."),
    ChatMessage(role=MessageRole.USER, content="How do I create an index?"),
    ChatMessage(role=MessageRole.ASSISTANT, content="LlamaIndex is a data framework for LLM-based applications which benefit from context augmentation."),
]

llm = OpenAI(model="gpt-4", temperature=0)

documents = SimpleDirectoryReader("data").load_data()
index = VectorStoreIndex.from_documents(documents)


engine = index.as_query_engine(llm=llm)

chat_engine = CondenseQuestionChatEngine.from_defaults(
    query_engine=engine,
    condense_question_prompt=custom_prompt,
    chat_history=chat_history,
    # verbose=True,
)

response = chat_engine.chat("What is LlamaIndex ? Who's Thiago ?")
# response = engine.chat("What is LlamaIndex ?")
print(response)

TT. Ventura

im trying to use gpt-4 to see if it gets smartter

WWhiteFang_Jr

So the process here is like this:
When you pass the query , it appends it inside the chat history and then form the new question. Which is passed to the query engine to query over your docs

WWhiteFang_Jr

If you want the llm to answer some answers, you can add it into the system prompt or maybe define the prompt for your llm queries and add these details over there.

WWhiteFang_Jr

https://docs.llamaindex.ai/en/stable/module_guides/models/prompts.html#prompts

TT. Ventura

I'm struggling a little to understand. What I'm trying to do is to mix both knowledge from LLMs and my documents. For example, if the user asks something about my CV LlamaIndex can answer based on my CV info but if the user asks something broader, as "What's LlamaIndex?" or "What is Python?" it should come from LLM base knowledge.

TT. Ventura

If I understood your suggestion well. I'm trying to change the prompt.

Plain Text

custom_prompt = PromptTemplate(
    """\
        Given a conversation (between Human and Assistant) and a follow up message from Human, \
        provide the information needed based on context and general knowledge from documents and LLMs

        <Chat History>
        {chat_history}

        <Follow Up Message>
        {question}
    """
)

but the answer is still bad.

Q:

"What is LlamaIndex ? Who's Thiago ?"

The context does not provide any information about LlamaIndex or its relation to Thiago Ventura. Therefore, I cannot provide any details about Thiago's involvement with LlamaIndex.

TT. Ventura

my bad. I think it's working. The problem is that GPT 4 does not know Llama Index yet

TT. Ventura

thanks for all the help

WWhiteFang_Jr

Yea, That's why said to put it into the system text or in prompt as GPT 4 is trained on knowledge upto 2021 Sept or dec

WWhiteFang_Jr

Or maybe create a document and add it to your data

TT. Ventura

I'm starting to enjoy Llama Index 🙂

WWhiteFang_Jr

Awesome!!

Add a reply

Find answers from the community

Chat engine