Hi Guys,

llancerninja

Hi Guys,

I have already built a solution to create custom AI chatbots from own data which uses gpt-3.5-turbo and RAG to fetch the context using openai embedding and cosine similarity match.

I am facing issues when user asks some follow up question on the previous question, in this case doing RAG for just the question is not sufficent as it will not fetch the required context.

I already tried question rephrasing based on conversation history but it makes responses slow and difficult to decide which one to rephrase and which one to not.

I went through the source code of chat llama_index and couldn't find anything different, follow up questions are not handled is what I observed. Can someone please help me on this

13 comments

LLogan M

The best option is probably just using an agent

But for this to work well, there will always be an extra LLM step to rephrase the question 🤔 Both the condense question chat engine and agents will do this, but I feel an agent is more flexible

llancerninja

I tried agents but it messes things up sometimes not too reliable, like when generic messages like "hi", "ok" are sent it does something crazy entirely messing up everything

llancerninja

I want to keep it conversational

llancerninja

The problem with condense question is that it rephrases every single input from user, how to decide which is a follow up question and which is not, if it rephrases everything messages like "hi", "ok" will be messed up

LLogan M

yea, use an agent

LLogan M

just need to write proper system prompts and tool descriptions

LLogan M

to tune it to your needs and data

LLogan M

Plain Text

query_engine_tools = [
    QueryEngineTool(
        query_engine=lyft_engine,
        metadata=ToolMetadata(
            name="lyft_10k",
            description=(
                "Provides information about Lyft financials for year 2021. "
                "Use a detailed plain text question as input to the tool."
            ),
        ),
    ),
]

agent = OpenAIAgent.from_tools(query_engine_tools, system_prompt="You are a conversational assistannt, capable of using tools to look up information about Lyft 2021 financial data.", verbose=True)

LLogan M

Something like that

LLogan M

with specific details

LLogan M

https://docs.llamaindex.ai/en/stable/examples/agent/openai_agent_with_query_engine.html

llancerninja

How many LLM call agent would do for a query, response time is important as well

llancerninja

I have tried https://chatbase.co and it works pretty well with follow up questions, and very fast, don't think they use agents

Add a reply

Find answers from the community

Hi Guys,