LlamaIndex
Log in

Find answers from the community

Updated 2 years ago

Wondering if there s any way to save

bbbornsztein
·
Wondering if there's any way to save/load an index to S3 or some other filestore? I suppose we'd then need to download the whole index every time a query is run?
j
b
11 comments
jjerryjliu0
this is a good idea. currently most indices are serializable to a json file on disk, and when loaded they're stored in memory
jjerryjliu0
some indices like weaviate/pinecone are backed by respective db's
bbbornsztein
right - i didn't want to mess with weviate for a json index, so I'm saving to disk
bbbornsztein
but my app is on fly.io and working with storage is tricky
bbbornsztein
it'd be better to be able to just store the indexes in s3
bbbornsztein
or what about in postgres?
jjerryjliu0
yep makes sense
bbbornsztein
that would be super eary
bbbornsztein
easy
jjerryjliu0
i want to add "serializer" classes so that you can save this stuff to external stores
bbbornsztein
yeah, makes sense
Add a reply
Sign up and join the conversation on Discord