Hi I m trying to save an additional

At a glance

Hi, I'm trying to save an additional index into the same persist directory, but it seems like the pre-existed index is replaced with te new index. Here's my code:

index.set_index_id("id1")
index.storage_context.persist(persist_dir="./test_storage")

index.set_index_id("id2")
index.storage_context.persist(persist_dir="./test_storage")

I can only load the index ("id2"), but not the first index ("id1"). The first one is deleted. How can I save the multiple index? Thanks!

9 comments

SSeldo

Have you tried using multiple persistence directories? (persist_dir="./test_storage_2")

MMrMJ

I can do this and this should work. But, document says we can save multiple indexes into the same directory too.

SSeldo

Sorry, don't know 🙂

LLogan M

@MrMJ each index needs to share the same storage context I think for this to work

Plain Text

storage_context = StorageContext.from_defaults()

index1 = VectorStoreIndex.from_documents(..., storage_context=storage_context)

index2 = VectorStoreIndex.from_documents(..., storage_context=storage_context)

LLogan M

Then calling persist with different IDs should work

MMrMJ

Thanks @Logan M !

It's working, but I'm a bit confused. Below is my current logic to save the two different index with different IDs and the same persisting dir:


index1 = VectorStoreIndex.from_documents(documents1, service_context=service_context)
index1.set_index_id(index_name1)
index1.storage_context.persist(persist_dir=persist_dir)

index2 = VectorStoreIndex.from_documents(documents2, service_context=service_context)
index2.set_index_id(index_name2)
index2.storage_context.persist(persist_dir=persist_dir)

How can I apply your solution to this logic?

LLogan M

Like this

Plain Text

from llama_index import StorageContext

storage_context = StorageContext.from_defaults()

index1 = VectorStoreIndex.from_documents(documents1, service_context=service_context, storage_context=storage_context)
index1.set_index_id(index_name1)
index1.storage_context.persist(persist_dir=persist_dir)

index2 = VectorStoreIndex.from_documents(documents2, service_context=service_context, storage_context=storage_context)
index2.set_index_id(index_name2)
index2.storage_context.persist(persist_dir=persist_dir)

MMrMJ

@Logan M This works perfectly 🙂 Thanks a lot! I didn't know 'storage_context' has to be defined first.

MMrMJ

@Logan M One follow-up question! Is there a way to efficiently save and load the storage_context? I'm asking this because I want to add the new index later. THX!

Add a reply

Find answers from the community

Hi I m trying to save an additional