Docstore

DDeval

Hi all, I am building llamaindex based RAG application.

I had few questions about how crud operations work in llamaindex docstore (redis) and qdranr

When user uploads a document after initial ingestion, how can i apply the transformation defined in the IngestionPipeline to create the nodes and insert in the existing index.

How can I manage data isolation ( per user) across docstore and vector db consistently? Any suggestions. I came across multi tenancy in qdrant but couldn’t confirm whether it handles isolation at docstore level

5 comments

LLogan M

Attach the vector store and docstore to the ingestion pipeline? (But also ensuring consistent input doc ids if inputting the same doc more than once)

Hmm. Might have to use a different namespace per user

DDeval

I did not understand.

If I implement multitenancy for qdrant db to isolate data per user, do I need to make changes to make the isolation consistent in docstore or is it automatically handled (i have defined both docstore and vector db in ingestion pipeline)?

LLogan M

I think you'd need to change the docstore namespace to be namespaced per user ID or similar

DDeval

Ok, so if I define the namespace in docstore and do payload filtering in qdrant db, I need to make sure that both docstore namespace and qdrant group id is mapped consistently, right? It is not automatically handled internally (the mapping)?

LLogan M

Yea not automatic 👍

Add a reply

Find answers from the community

Docstore