Docstore

At a glance

The community member is building a llamaindex-based RAG (Retrieval Augmented Generation) application and has questions about CRUD operations in the llamaindex docstore (Redis) and Qdrant. Specifically, they want to know how to apply the transformations defined in the IngestionPipeline when a user uploads a document after the initial ingestion, and how to manage data isolation (per user) across the docstore and vector database consistently.

The comments suggest that the community member should attach the vector store and docstore to the ingestion pipeline, ensuring consistent input document IDs. They also suggest using a different namespace per user for the docstore. Regarding multi-tenancy in Qdrant, the community members discuss whether it handles isolation at the docstore level, and conclude that the mapping between the docstore namespace and Qdrant group ID needs to be managed consistently, as it is not automatically handled internally.

There is no explicitly marked answer in the provided information.

DDeval

Hi all, I am building llamaindex based RAG application.

I had few questions about how crud operations work in llamaindex docstore (redis) and qdranr

When user uploads a document after initial ingestion, how can i apply the transformation defined in the IngestionPipeline to create the nodes and insert in the existing index.

How can I manage data isolation ( per user) across docstore and vector db consistently? Any suggestions. I came across multi tenancy in qdrant but couldn’t confirm whether it handles isolation at docstore level

5 comments

LLogan M

Attach the vector store and docstore to the ingestion pipeline? (But also ensuring consistent input doc ids if inputting the same doc more than once)

Hmm. Might have to use a different namespace per user

DDeval

I did not understand.

If I implement multitenancy for qdrant db to isolate data per user, do I need to make changes to make the isolation consistent in docstore or is it automatically handled (i have defined both docstore and vector db in ingestion pipeline)?

LLogan M

I think you'd need to change the docstore namespace to be namespaced per user ID or similar

DDeval

Ok, so if I define the namespace in docstore and do payload filtering in qdrant db, I need to make sure that both docstore namespace and qdrant group id is mapped consistently, right? It is not automatically handled internally (the mapping)?

LLogan M

Yea not automatic 👍

Add a reply

Find answers from the community

Docstore