Find answers from the community

Updated 5 months ago

how can I load multiple indices as a retriever?

At a glance

The community members are discussing how to load multiple indices as a retriever in LlamaIndex. The main options presented are:

1. Using the QueryFusionRetriever class to fuse multiple indexes into a single retriever, which can also generate additional queries and aggregate the results.

2. Using the RecursiveRetriever class, where the community members define separate IndexNodes for each chunk size, put them into a SummaryIndex, and then create the RecursiveRetriever with the summary index as the root.

The community members are also discussing how to create the RecursiveRetriever from existing VectorStoreIndex objects, but there is no clear answer provided.

how can I load multiple indices as a retriever?
k
J
S
6 comments
Please stand by @Jam Mirzakhalov as I search the knowledge base. I normally take about 20 seconds or so. :wave:
----------------------
You can load multiple indices as a retriever by using the QueryFusionRetriever or RecursiveRetriever classes in LlamaIndex.

For QueryFusionRetriever, you can fuse your indexes into a single retriever. This retriever will also generate extra queries related to the original question, and aggregate the results. Here is an example of how to use it:

Plain Text
from llama_index.retrievers import QueryFusionRetriever

retriever = QueryFusionRetriever(
    [index_1.as_retriever(), index_2.as_retriever()],
    similarity_top_k=2,
    num_queries=4,  # set this to 1 to disable query generation
    use_async=True,
    verbose=True,
    # query_gen_prompt="...", # we could override the query generation prompt here
)

Source ()

For RecursiveRetriever, you can define a separate IndexNode corresponding to the vector retriever for each chunk size, put all IndexNodes into a single SummaryIndex, and then define a Recursive Retriever, with the root node being the summary index retriever. Here is an example of how to use it:

```python
from llama_index.tools import RetrieverTool
from llama_index.schema import IndexNode
from llama_index.retrievers import RecursiveRetriever
from llama_index import SummaryIndex

retriever_dict = {}
retriever_nodes = []
for chunk_size, vector_index in zip(chunk_sizes, vector_indices):
nodeid = f"chunk{chunk_size}"
node = IndexNode(
text=(
"Retrieves relevant context from the Llama 2 paper (chunk size"
f" {chunk_size})"
),
index_id=node_id,
)
retriever_nodes.append(node)
retriever_dict[node_id] = vector_index.as_retriever()

summary_index = SummaryIndex(retriever_nodes)
retriever = RecursiveRetriever(
root_id="root",
retriever_dict={"root": summary_index.as_retriever(), retriever_dict},)```Source** ()

Please replace index_1, index_2, chunk_sizes, and vector_indices with your actual indices and chunk sizes.
----------------------
Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:
----------------------
@kapa.ai I have already loaded indices where each object is VectorStoreIndex. How do I get the nodes and create a Recursive Retriever
@kapa.ai I have already loaded indices where each object is VectorStoreIndex. How do I get the nodes and create a Recursive Retriever

index_objects = self.initialize_multiple_indexes(indices=indices)
@Jam Mirzakhalov I am stuck at this same spot on how to send the nodes dictionary to node_dict of RecursiveRetriever from an existing VectorStoreIndex. Were you able to figure this out?
Add a reply
Sign up and join the conversation on Discord