Find answers from the community

Updated 3 months ago

Hello all πŸ‘‹ I am wondering if I can

Hello all πŸ‘‹ I am wondering if I can feed all my data (pdf, docs, txt,...) to LlamaIndex. Can it handle and deal 500GB of data?
W
R
6 comments
500GB is HUGE!! I would recommend using vector store,
Keeping 500GB of data in memory is very risky.
https://docs.llamaindex.ai/en/stable/module_guides/storing/vector_stores.html
Take a look at the available intergrations for Vector stores in Llamaindex
Yeah I know, it seems huge to me too πŸ˜…
Am searching everywhere if there is a limitation but I don't find the info.
Are Vector stores limited in size?
There is no limitation from llamaindex side, Only limitation would be your memory constraint πŸ˜… .

No There should be no size constraint at vector stores. Only thing that I could think of would be that in free version they might give let say 50GB and beyond that they may ask you to buy pro version.
Thank @WhiteFang_Jr that makes a lot of sense πŸ™‚
Add a reply
Sign up and join the conversation on Discord