Find answers from the community

Updated 4 months ago

storing multiple indexes locally

At a glance
it looks like storing multiple indexes locally in the same folder is unsupported. despite defining multiple index ids, it looks like only the last written persisted index "wins"
t
L
6 comments
Plain Text
webconsole_documents = SimpleDirectoryReader('openshift-docs', file_metadata=filename_fn).load_data()
webconsole_index = VectorStoreIndex.from_documents(webconsole_documents, service_context=service_context)
webconsole_index.set_index_id("docs")
webconsole_index.storage_context.persist("./vector-db")

summary_documents = SimpleDirectoryReader('summary-docs', file_metadata=filename_fn).load_data()
summary_index = VectorStoreIndex.from_documents(summary_documents, service_context=service_context)
summary_index.set_index_id("summary")
summary_index.storage_context.persist("./vector-db")


When I go to query later:

Plain Text
storage_context = StorageContext.from_defaults(persist_dir="./vector-db")
index = load_index_from_storage(storage_context=storage_context, index_id="docs", service_context=service_context)


the query fails:

Plain Text
Traceback (most recent call last):
  File "/home/thoraxe/Red_Hat/openshift/llamaindex-experiments/ops-sop-chatbot/pdf_query.py", line 48, in <module>
    index = load_index_from_storage(storage_context=storage_context, index_id="docs", service_context=service_context)
  File "/home/thoraxe/.pyenv/versions/llamaindex-39/lib/python3.9/site-packages/llama_index/indices/loading.py", line 33, in load_index_from_storage
    indices = load_indices_from_storage(storage_context, index_ids=index_ids, **kwargs)
  File "/home/thoraxe/.pyenv/versions/llamaindex-39/lib/python3.9/site-packages/llama_index/indices/loading.py", line 71, in load_indices_from_storage
    raise ValueError(f"Failed to load index with ID {index_id}")
ValueError: Failed to load index with ID docs
They need to share the same storage context object when creating the initial indexes
Plain Text
storage_context = StorageContext.from_defaults()

index1 = VectorStoreIndex.from_documents(..., storage_context=storage_context)
index1.set_index_id("index1")

index2 = VectorStoreIndex.from_documents(..., storage_context=storage_context)
index2.set_index_id("index2")

storage_context.persist(persist_dir="./storage")
that makes snese
ok that did the trick, ty
Add a reply
Sign up and join the conversation on Discord