Can I merge multiple vector store index

Can I merge multiple vector store index that built previously?

5 comments

Yeah you can. Just take the nodes from other indexes and add it to the one you want to keep.

Plain Text

nodes1 = index1.docstore.docs # Extract nodes from the existing index

for node in nodes1:
     final_index.insert(node)

## ONce all the nodes are added, persist the index.

PS: This will only work if you have the nodes in your docstore

RRuby

Thanks!! But the nodes are empty by using this method, and I checked the chromadb by retrieving, they are not empty. Here is my code:

Plain Text

# load db
db = chromadb.PersistentClient(path=db_path1)
chroma_collection = db.get_or_create_collection("quickstart")
vector_store = ChromaVectorStore(chroma_collection=chroma_collection)
index = VectorStoreIndex.from_vector_store(vector_store, service_context=ServiceContextModel()())

# load db_final
db_final = chromadb.PersistentClient(path=db_path2)
chroma_collection_final = db_final.get_or_create_collection("quickstart")
vector_store_final = ChromaVectorStore(chroma_collection=chroma_collection_final)
index_final = VectorStoreIndex.from_vector_store(vector_store_final, service_context=ServiceContextModel()())

nodes = index.docstore.docs
for node in nodes:
    index_final.insert(node)

WWhiteFang_Jr

There was a arg called store_node_override or something. That would fetch all the nodes back to docstore. I may be wrong

WWhiteFang_Jr

But since you have the chroma client with you, fethc all the stored nodes from the DB using client and then you can extract the text and metadata, create a Document and insert it into your final index

RRuby

OK, I got it, thank you☺️

Add a reply

Find answers from the community

Can I merge multiple vector store index