Hello everyone, I am working on a RAG project where I b...

HHK

@kapa.ai Hello everyone, I am working on a RAG project where I built a very simple pipeline with OpenAI embeddings (with no modules from Llamaindex) and doing simple embedding similarity and it is working good but there are a few problems such as it is not asking follow-up questions to have more precise results and I think that's where I will need to implement agents. I have been reading LlamaIndex documentation and I read about Controllable agents which can be used to ask followup questions but still there are a number of questions and confusions about how to use Controllable agents in such a way that it should ask some relevant followup question before doing retrieval.

Would it be enough to just add a tool def ask_follow_up_question and let the agent think about asking a question or doing the retrieval search? I know this is not an easy question without knowing the context and the problem I am working on so here is a bit more details;

Let's suppose you are having a furniture shop where you are selling bed, sofa, etc. user is asking a query "Do you have any king size bed?"
Then the answer should be "Yes" but a few followup questions can be ["Do you have any budget in mind", "Do you have any color preferences?", "What kind of material are you looking for?", etc]

So how can I implement an agent like this? Any help would be appreciated and you can point me towards some useful links as well. Thanks 🙂

10 comments

kkapa.ai

Please stand by @HK as I search the knowledge base. I normally take about 20 seconds or so. :wave:

kkapa.ai

----------------------
Hello! It sounds like you're looking to add more interactivity to your RAG pipeline by using agents to ask follow-up questions. This is indeed a great way to refine the results and provide more precise answers.

LlamaIndex provides a lower-level agent API that allows controllable step-wise execution on top of a RAG pipeline. This can be used to add agentic capabilities to your pipeline, which can help in reasoning over more complex questions. The lack of steerability/transparency is a common issue with agents, but with LlamaIndex's agent API, you can control the execution step-by-step, which can be useful in your case to ask follow-up questions.

Here's a basic example of how you can set up an OpenAI Agent using LlamaIndex:

Plain Text

from llama_index.core.agent import AgentRunner
from llama_index.agent.openai import OpenAIAgentWorker, OpenAIAgent
from llama_index.agent.openai import OpenAIAgentWorker

openai_step_engine = OpenAIAgentWorker.from_tools(
    query_engine_tools, llm=llm, verbose=True
)
agent = AgentRunner(openai_step_engine)
# # alternative
# agent = OpenAIAgent.from_tools(query_engine_tools, llm=llm, verbose=True)

(Source: Jerryjliu > Llama_index > Blob > Main > Docs > Docs > Examples > Agent > Agent_runner > Controllable Agents for RAG > Setup Agent > Setup OpenAI Agent ())

However, the specific implementation of asking follow-up questions would depend on your application logic. You might need to design your agent to understand the context of the conversation and decide when to ask a follow-up question. This could involve designing a decision-making process within your agent to determine when a follow-up question is necessary, and what kind of question to ask.

kkapa.ai

For more examples and guides on using agents with LlamaIndex, you can refer to the following links:

Agent Runner ()
Agent Runner RAG ()
Controllable Agent Runner ()

I hope this helps! Let me know if you have any more questions.
----------------------
Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:
----------------------

HHK

@kapa.ai can you take the given example in my question and define a simple RAG implementation with agents so that I can further enhance your implementation.

kkapa.ai

Let me check the knowledge base for you. One moment! :books:

kkapa.ai

----------------------
I'm sorry, but based on the provided knowledge sources, I don't have enough information to provide a detailed implementation of a RAG system with agents for your specific use case. However, I can provide a general outline of how you might approach this problem using LlamaIndex.

Setup Data: Load your data (in your case, information about furniture) using a suitable reader. For example, if your data is stored in a directory, you can use the SimpleDirectoryReader.

Plain Text

from llama_index.core import SimpleDirectoryReader

reader = SimpleDirectoryReader()
docs = reader.load_data(directory_path="path_to_your_data")

Setup LLM and Service Context: Initialize your language model and service context. In this case, we're using OpenAI's GPT-4 model.

Plain Text

from llama_index.core import ServiceContext
from llama_index.llms.openai import OpenAI
from llama_index.core.callbacks import CallbackManager

llm = OpenAI(temperature=0, model="gpt-4")
service_context = ServiceContext.from_defaults(llm=llm)
callback_manager = CallbackManager([])

Setup Query Engine: Create an embedding store and index, and then set up your query engine.

Plain Text

from llama_index.core import VectorStoreIndex

index = VectorStoreIndex.from_documents(documents, transformations=[splitter], embed_model=embed_model)
query_engine = index.as_query_engine(llm=llm)

Setup Agent: Initialize your agent. This is where you would add your custom logic for asking follow-up questions.

Plain Text

from llama_index.core.agent import AgentRunner
from llama_index.agent.openai import OpenAIAgentWorker

openai_step_engine = OpenAIAgentWorker.from_tools(query_engine_tools, llm=llm, verbose=True)
agent = AgentRunner(openai_step_engine)

kkapa.ai

Run Queries: Now you can run queries using your agent. The agent should be able to handle complex queries and ask follow-up questions as needed.

Plain Text

response = agent.chat("Do you have any king size bed?")

Please note that this is a very high-level outline and the actual implementation will depend on your specific requirements and data. For more detailed examples and guides, you can refer to the LlamaIndex documentation ().
----------------------
Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:
----------------------

HHK

@kapa.ai I have all the documents and their embeddings in the supabase so there is no need to create the embeddings again. Write supabase vectorstore retrieval

kkapa.ai

Let me check the knowledge base for you. One moment! :books:

kkapa.ai

----------------------
Sure, if you already have your documents and their embeddings stored in Supabase, you can use the SupabaseVectorStore to retrieve the embeddings. Here's a basic example of how you can set up a SupabaseVectorStore and perform a retrieval:

First, you need to install the necessary packages:

Plain Text

%pip install llama-index-vector-stores-supabase
!pip install llama-index

Then, you can set up the SupabaseVectorStore:

Plain Text

from llama_index.vector_stores.supabase import SupabaseVectorStore
from llama_index.core import StorageContext, VectorStoreIndex

vector_store = SupabaseVectorStore(
    postgres_connection_string="postgresql://<user>:<password>@<host>:<port>/<db_name>",
    collection_name="base_demo",
)

storage_context = StorageContext.from_defaults(vector_store=vector_store)

Now, you can use the VectorStoreIndex to perform a retrieval:

Plain Text

index = VectorStoreIndex.from_documents(documents, storage_context=storage_context)

Please replace <user>, <password>, <host>, <port>, and <db_name> with your actual Supabase credentials.

For more details, you can refer to the Supabase Vector Store example () and the SupabaseVectorStore API reference ().
----------------------
Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:
----------------------

Add a reply

Find answers from the community

Hello everyone, I am working on a RAG project where I built a very simple pipeline with Op