Hello all 👋 I am wondering if I can

500GB is HUGE!! I would recommend using vector store,
Keeping 500GB of data in memory is very risky.
https://docs.llamaindex.ai/en/stable/module_guides/storing/vector_stores.html

Take a look at the available intergrations for Vector stores in Llamaindex

Yeah I know, it seems huge to me too 😅
Am searching everywhere if there is a limitation but I don't find the info.

Are Vector stores limited in size?

There is no limitation from llamaindex side, Only limitation would be your memory constraint 😅 .

No There should be no size constraint at vector stores. Only thing that I could think of would be that in free version they might give let say 50GB and beyond that they may ask you to buy pro version.

Thank @WhiteFang_Jr that makes a lot of sense 🙂