Overwrite

Not really easily 😅 you'd have to use the refresh feature, which depends on having the docstore (activated with an override) and having consistent doc_ids on input documents.

Then, when a documents content has been updated, it's nodes will be overwritten

Here's an example with chroma. Postgres would be very similar

https://discord.com/channels/1059199217496772688/1163880111074971790/1163900056718553169

Thanks Logan. Follow up question. Why do we need index storage if we use Vector store to store indices?

Because if you want to support refresh functionality, there needs to be a layer on top of the vector db tracking what's been inserted (I.e. the docstore)

Gotcha.

The example you shared does overwrite the index and document storage, but in vectordb, it appears to be appending rather than overwriting. Is this a default pattern. If so, is there currently a way to overwrite the rows in vectordb?

In the example it should be appending, unless the document is already inserted, then it's an upsert 🤔 But it relies on the doc_id of the inputs being consistent (i.e. the same source document should have the same doc_id, even if it's content has changed)

ahh okay. There was an error on my end. it does indeed upsert. Thanks a lot for the help!

Feature Request: Ability to update node by ID

But why 👀 documents are the main thing loaded/ingested

A small hack would be setting the parent of the node to the same as the node ID

Because I have a complicated IngestionPipeline that occasionally fails (but continues so as to not fail very large batches), and now I need to update those failing nodes' metadata to match the pattern in the rest of the data

(and only a subset of a document's nodes fail, so I don't want to reload the entire document)