Hi I m trying to save an additional

At a glance

The community member is trying to save multiple indexes in the same persist directory, but the previous index is being replaced by the new one. They have tried setting different index IDs, but can only load the latest index. A community member suggests using multiple persistence directories, but the original poster says the documentation indicates it should be possible to save multiple indexes in the same directory.

Another community member suggests that each index needs to share the same storage context for this to work. They provide an example of how to set up the storage context and create the indexes, which the original poster confirms works. However, the original poster has a follow-up question about efficiently saving and loading the storage context, as they want to add new indexes later.

There is no explicitly marked answer, but the community members have provided a solution that involves using a shared storage context when creating the indexes and persisting them to the same directory.

MMrMJ

Hi, I'm trying to save an additional index into the same persist directory, but it seems like the pre-existed index is replaced with te new index. Here's my code:

index.set_index_id("id1")
index.storage_context.persist(persist_dir="./test_storage")

index.set_index_id("id2")
index.storage_context.persist(persist_dir="./test_storage")

I can only load the index ("id2"), but not the first index ("id1"). The first one is deleted. How can I save the multiple index? Thanks!

9 comments

SSeldo

Have you tried using multiple persistence directories? (persist_dir="./test_storage_2")

MMrMJ

I can do this and this should work. But, document says we can save multiple indexes into the same directory too.

SSeldo

Sorry, don't know 🙂

LLogan M

@MrMJ each index needs to share the same storage context I think for this to work

Plain Text

storage_context = StorageContext.from_defaults()

index1 = VectorStoreIndex.from_documents(..., storage_context=storage_context)

index2 = VectorStoreIndex.from_documents(..., storage_context=storage_context)

LLogan M

Then calling persist with different IDs should work

MMrMJ

Thanks @Logan M !

It's working, but I'm a bit confused. Below is my current logic to save the two different index with different IDs and the same persisting dir:


index1 = VectorStoreIndex.from_documents(documents1, service_context=service_context)
index1.set_index_id(index_name1)
index1.storage_context.persist(persist_dir=persist_dir)

index2 = VectorStoreIndex.from_documents(documents2, service_context=service_context)
index2.set_index_id(index_name2)
index2.storage_context.persist(persist_dir=persist_dir)

How can I apply your solution to this logic?

LLogan M

Like this

Plain Text

from llama_index import StorageContext

storage_context = StorageContext.from_defaults()

index1 = VectorStoreIndex.from_documents(documents1, service_context=service_context, storage_context=storage_context)
index1.set_index_id(index_name1)
index1.storage_context.persist(persist_dir=persist_dir)

index2 = VectorStoreIndex.from_documents(documents2, service_context=service_context, storage_context=storage_context)
index2.set_index_id(index_name2)
index2.storage_context.persist(persist_dir=persist_dir)

MMrMJ

@Logan M This works perfectly 🙂 Thanks a lot! I didn't know 'storage_context' has to be defined first.

MMrMJ

@Logan M One follow-up question! Is there a way to efficiently save and load the storage_context? I'm asking this because I want to add the new index later. THX!

Add a reply

Find answers from the community

Hi I m trying to save an additional