M

·

Hi, is there an example somewhere of

Hi, is there an example somewhere of using any external vector store (pinecone, milvus, etc.) with stores_text=False and using an external document store (Mongo, redis, etc.)?
A lot node data is not used by the vector store for search and the resource cost of most vector stores, so this seems like it should be a standard pattern for scaling, but I can't find much documentation on it or any examples.

5 comments

M

L

MM

·

Hi!

Hi!

I'm having an issue with the milvus vector store. The delete function contains this code:

Plain Text

        entries = self._milvusclient.query(
            collection_name=self.collection_name,
            filter=f"{self.doc_id_field} in [{','.join(doc_ids)}]",
        )
        if len(entries) > 0:
            ids = [entry["id"] for entry in entries]
         self._milvusclient.delete(collection_name=self.collection_name, pks=ids)
            logger.debug(f"Successfully deleted embedding with doc_id: {doc_ids}")

The problem is that the first line can fail with grpc: received message larger than max . I have documents with over 1,000 chunks, so I often run into this even when deleting a single document.

Is there a reason why this code is written this way instead of this:

Plain Text

    res = self._milvusclient.delete(
        collection_name=self.collection_name,
        filter=f"{self.doc_id_field} in [{','.join(doc_ids)}]",
    )
    logger.debug(f"Successfully deleted {res.get('delete_count', 0)} nodes from the vector store")

?

3 comments

M

L

Find answers from the community

Hi, is there an example somewhere of

Hi!