Find answers from the community

Updated 3 months ago

storing multiple indexes locally

it looks like storing multiple indexes locally in the same folder is unsupported. despite defining multiple index ids, it looks like only the last written persisted index "wins"
t
L
6 comments
Plain Text
webconsole_documents = SimpleDirectoryReader('openshift-docs', file_metadata=filename_fn).load_data()
webconsole_index = VectorStoreIndex.from_documents(webconsole_documents, service_context=service_context)
webconsole_index.set_index_id("docs")
webconsole_index.storage_context.persist("./vector-db")

summary_documents = SimpleDirectoryReader('summary-docs', file_metadata=filename_fn).load_data()
summary_index = VectorStoreIndex.from_documents(summary_documents, service_context=service_context)
summary_index.set_index_id("summary")
summary_index.storage_context.persist("./vector-db")


When I go to query later:

Plain Text
storage_context = StorageContext.from_defaults(persist_dir="./vector-db")
index = load_index_from_storage(storage_context=storage_context, index_id="docs", service_context=service_context)


the query fails:

Plain Text
Traceback (most recent call last):
  File "/home/thoraxe/Red_Hat/openshift/llamaindex-experiments/ops-sop-chatbot/pdf_query.py", line 48, in <module>
    index = load_index_from_storage(storage_context=storage_context, index_id="docs", service_context=service_context)
  File "/home/thoraxe/.pyenv/versions/llamaindex-39/lib/python3.9/site-packages/llama_index/indices/loading.py", line 33, in load_index_from_storage
    indices = load_indices_from_storage(storage_context, index_ids=index_ids, **kwargs)
  File "/home/thoraxe/.pyenv/versions/llamaindex-39/lib/python3.9/site-packages/llama_index/indices/loading.py", line 71, in load_indices_from_storage
    raise ValueError(f"Failed to load index with ID {index_id}")
ValueError: Failed to load index with ID docs
They need to share the same storage context object when creating the initial indexes
Plain Text
storage_context = StorageContext.from_defaults()

index1 = VectorStoreIndex.from_documents(..., storage_context=storage_context)
index1.set_index_id("index1")

index2 = VectorStoreIndex.from_documents(..., storage_context=storage_context)
index2.set_index_id("index2")

storage_context.persist(persist_dir="./storage")
that makes snese
ok that did the trick, ty
Add a reply
Sign up and join the conversation on Discord