Find answers from the community

Updated 2 years ago

Hey a quick question on persistent

Hey, a quick question on persistent storage. To store the indices and graphs that are generated, should I be using a database or an S3 storage? I have tried using S3, but I am getting errors with serialising and deserialising.

Thanks in advance for any tips! 🙏

8 comments

cchanansh

maybe check StorageContext

cchanansh

https://gpt-index.readthedocs.io/en/latest/how_to/storage/save_load.html

ddisiok

@chanansh thanks for helping answer

ddisiok

@pineapple right now we support directly saving into MongoDB, should be very easy to add S3

ddisiok

will get to it in the near future, also open to contribution

cchanansh

I am new here. Happy to contribute. Still figuring why parsing one document takes so long.

ddisiok

how long is your document and how long is it taking?

ddaemon

here's a first pass at S3 support: https://github.com/jerryjliu/llama_index/pull/3350

Add a reply