I m a bit confused about how the

LLLYX

I'm a bit confused about how the interactions between indices, documents, and vector stores work. I'm using QDrant atm. So I see from QdrantReader that it basically just assumes you're adding a payload keyed with 'text' to retrieve the documents from. For the StorageContext, how would I specify that I want to use Qdrant as the Document Store?

And to clarify, Qdrant would be able to store the vectors, the documents (kinda, by hacking the payload feature), but to store the created indices, would have to use something like S3? So the update/modify path would be basically insert new document into Qdrant, and then load the associated index from s3, do the insertion separately there, and re-save onto s3?

15 comments

LLogan M

As far as I know, QDrant (and most vector stores) store the entire index on the vector_store (a little hacky, but it's extremely convienant)

To re-connect to a vector store you created on QDrant, you should be able to do something like this (assuming you connect the client to a previously created vector store on qdrant)

Plain Text

vector_store = QdrantVectorStore(client=client, collection_name="paul_graham")
storage_context = StorageContext.from_defaults(vector_store=vector_store)
index = VectorStoreIndex([], storage_context=storage_context)

LLLYX

So I shouldn't need to set a IndexStore in the StorageContext?

LLogan M

nope

LLogan M

it wont' be used even if you set it 😅

LLogan M

(as far as I know)

LLogan M

Basically with a vector store integration, you don't really need to worry about calling persist()

LLogan M

You can see here, that we don't actually update the index_struct or docstore https://github.com/jerryjliu/llama_index/blob/ddf7ff875796c7879a62decc8ac68149e7328bd9/llama_index/indices/vector_store/base.py#L160

LLLYX

Ok, and can I assume then the docs will also be stored with the corresponding text already chunked in the vector store?

LLLYX

So when I retrieve the text using the reader, it'll correspond to the chunk of the original unchunked text that the vector was made from?

LLogan M

Yea, the text is also stored in the vector_store as well (for qdrant, it's part of the payload metadata dict)

LLogan M

Curious what the usecase is for the reader? I wasn't sure if anyone even used those lol

LLLYX

Hopefully won't need it now, but was thinking of cases when I might need to rebuild the index from the docs without having the users upload everything again

LLogan M

ah, I see! Hopefully this approach works for you then

LLLYX

Thanks! I'll probably be back with more questions lol, many changes from 0.5.x

LLogan M

Big upgrade lol! Sounds good :BOUNCE:

Add a reply

Find answers from the community

I m a bit confused about how the