I'm loading an index from disk like this

At a glance

The community member is loading an index from disk and wants to obtain the nodes to create a node_dict to pass into the RecursiveRetriever constructor. In the comments, another community member suggests accessing the nodes through index.docstore.docs.values(). However, when the community member switches to using ChromaDB, they are unsure how to access the nodes. The community members discuss potential workarounds, such as setting a high top-k value or using the raw ChromaDB client. They also discuss the practicality of using ChromaDB with RecursiveRetriever and maintaining a separate document store for the nodes. The community members suggest using other database libraries like Redis, MongoDB, or Firestore, and provide an example of using the SimpleDocumentStore from the llama_index library. However, there is no explicitly marked answer on how to access the nodes when using Qdrant Vector Store.

kknearest

I'm loading an index from disk like this:

Plain Text

index = load_index_from_storage(storage_context, service_context=service_context)

I want to obtain the nodes so that I can create a node_dict to pass into the RecursiveRetriever constructor:

Plain Text

retriever_chunk = RecursiveRetriever(
    "vector",
    retriever_dict={"vector": vector_retriever_chunk},
    node_dict=all_nodes_dict,
    verbose=True,
)

Is there a way to do this?

11 comments

LLogan M

nodes = index.docstore.docs.values()

kknearest

Thanks!

kknearest

@Logan M Now I'm using chromadb, so I load my index like this:

Plain Text

    index = VectorStoreIndex.from_vector_store(
        vector_store,
        service_context=service_context,
    )

How do I access the nodes in this case?

LLogan M

oof, a little harder. You could try setting the top-k to something huge and retrieving. Or using the raw chroma-client to fetch stuff

kknearest

Is it not practical to use chromadb with RecursiveRetriever?

LLogan M

You can, but it seems like youd have to maintain/keep track of a seperate docstore of nodes

kknearest

Is there another db library that works with RecursiveRetriever? If not, are there any examples of maintaining a separate docstore of nodes?

LLogan M

Plain Text

from llama_index.storage.docstore import SimpleDocumentStore

docstore = SimpleDocumentStore()
docstore.add_documents(documents)

docstore.persist(persist_path="./docstore.json")
new_docstore = SimpleDocumentStore.from_persist_path("./docstore.json")

LLogan M

there's also a few other versions (redis, mongodb, firestore)

kknearest

Thanks!

ddhiraj

I am trying to achieve this using Qdrant Vector Store, is it possible to create all_nodes_dict after loading it from the persisted vector store?

Add a reply

Find answers from the community

I'm loading an index from disk like this