Find answers from the community

Updated 2 months ago

Strange behavior with locally persisted DuckDB vector store

I am using locally persisted DuckDB vector store(https://docs.llamaindex.ai/en/stable/examples/vector_stores/DuckDBDemo/) for my RAG app. I am observing two strange behavior: 1) If i add 74KB file into a persisted duckdb file, it increase the file of updated duckdb file by 5MB and on a second instance it increase the existing duckdb file (950MB) by 300 MB by just adding 3 more PDFs of 5MB each. 2) If i delete the newly added docs, the edited duckdb file size doesn't goes back to original size, in fact it stays the same as it was before deletion. I'm using .add and .delete methods to add(nodes) or delete(docs_ids). Can anybody provide any hint? Thanks.
L
1 comment
I feel like this is specific to how duckdb stores vectors and text

the size of the files doesn't matter really -- what matters is how many vectors/nodes that file creates
Add a reply
Sign up and join the conversation on Discord