Find answers from the community

Updated 2 months ago

Transfer to chroma

I have crated a large VectorStoreIndex which i have persisted - almost 7 gb. I want to transfer this data into a ChromaDB, but do not see a good way to access transfer to Chroma automatically, or access the underlying nodes and embeddings in order to add them manually. Does anyone have a solution to this?
L
R
5 comments
Not really a built in way to do this, but I did suggest a slightly hacky method for this here

https://discord.com/channels/1059199217496772688/1123688066633191444/1123690468656566402
It might work! But untested! Lol
Awesome - it works. Posted solution on github issue as well. Would not be bad idea to have a "retreive nodes" function on any Index to easily reverse and create different database types.
As a note. As a simple persisted database, it was over 9gb of data all together with docstore and vectors saved as JSON. It took almost 1,5 minutes to do similarity search. With Chroma, it is 1,4gb, and is under 1 second. Should have a note in the docs that the simple index is not intended for significant amounts of data.
Glad it works, nice!! πŸ’ͺ

Also that sounds about right πŸ˜… since everything in the simple index is loaded into memory, and it just does a basic pairiwise comparison.
Add a reply
Sign up and join the conversation on Discord