Weaviate

This looks like that Weaviate is only allowing that many nodes to be fetched at once.

You could debug the code in this file for Weaviate
https://github.com/run-llama/llama_index/blob/f64551e906d3d64c8ff75ec3b9573b5011f182b5/llama_index/vector_stores/weaviate.py#L190

and check about how many records it is fetching in single turn.

Hi @WhiteFang_Jr, the query in the delete function by default returns 100 objects, which are being deleted, ideally it should delete all the instances of that particular doc_ref_id

Can Llama_index introduce the limit wherein the user will have the option to query and get the particular number of return objects and this limit will also make delete function more robust as well

Yes it should but since the vector store is being maintained by Weaviate and they may have set the limitation for fetching records upto 100 at once.
In-House vector stores delete every node related to that doc_id at once.

Does Weaviate provides such feature?

Yes, Using direct delete api through weaviate client, one can directly delete all the objects without any limit restriction

import weaviate.classes as wvc

collection = client.collections.get("EphemeralObject")
collection.data.delete_many(
where=wvc.Filter("name").like("EphemeralObject*")
)

In query part of delete function in Llama_index we can use .with_limit() to pass the limit

I'm not familiar with weaviates exact API, but this is the source code for delete

https://github.com/run-llama/llama_index/blob/59e24df3988f7b9721b72d33dd0f705623b57e40/llama_index/vector_stores/weaviate.py#L191

Maybe raise a PR if you spot something odd 🤔

Yep, Actually they are fetching records based on the doc ID and then calling the delete func.

Hey @Logan M @WhiteFang_Jr , I have created a PR for the same by including the "with_limit(10000)" function, it allows the deleting of 10000 object at once. One cannot delete objects more than 10000 without explicitly mentioning it in the weaviate deployment. You have to set " QUERY_MAXIMUM_RESULTS" variable in the weaviate config file while deploying it.