The community member is new and wants to build a search engine like a QnA bot that can handle various file types (PDF, TXT, etc.) uploaded by users. The main challenge is how to handle a large amount of data (e.g., 1 GB per user) without loading the full index into memory on the server.
A community member suggests using a shared bot between users, providing an example using Flask and the GPT Index library. However, another community member notes that the example only deals with a single file type, not a mix or combination of file types.
In response, another community member suggests using the SimpleDirectoryReader to handle various file types, but the files would need to be saved to disk temporarily and the reader pointed to them.
There is no explicitly marked answer in the comments.
Hello, New here I want to build a search engine like a QnA bot a user can have files like pdf, txt, words etc. Then he/she could question and bot should be able to answer and specify source files etc for more details, how can i achieve this as many users be uploading files a cannot load full index in memory to server if for example each user has 1 GB of data for n number of users.
@hammad Using the SimpleDirectoryReader, most combinations of files will work fine. Just have to save the files to disk temporarily and point the reader to them
