LlamaIndex
Log in

Find answers from the community

Updated 8 months ago

How does the Raptor Retriever in

nno_dice
·
How does the Raptor Retriever in LlamaIndex handle updates to a corpus of data? More specifically in this scenario:
  • I have persisted and clustered data in a vector index that was created previously via RAPTOR
  • I want to ADD more data to this persisted and clustered vector store, but I want to be sure this new data is included in the existing clusters
Does RAPTOR accomodate at all for step 2? Or do I need to truncate that table and just re-cluster the data to ensure the data is all clustered together?
L
n
6 comments
LLogan M
it does not. In fact, the way raptor works, if there is data added, you need to rebuild the entire index from scratch basically
LLogan M
Since the clusters and the summaries of clusters will change
nno_dice
I see - that makes sense. Thank you actually, in some ways thats not ideal but overall it simplifies things.
LLogan M
Yea I agree its not ideal -- if you are indexing over a large dataset that changes often, then raptor is pretty resource intensive
nno_dice
It is.
nno_dice
That's an interesting problem to solve though.
Add a reply
Sign up and join the conversation on Discord