Find answers from the community

Updated 2 years ago

Is there a straight forward way to load

OOrion Pax

Is there a straight forward way to load files from multiple directories into the GPTVectorStoreIndex?

6 comments

OOrion Pax

I currently have this, which works for 1 directory, but I want to combine multiple directories now:

Plain Text

persist_dir=f"/az/indexes/{index_id}"
service_context = _create_service_context()
documents = SimpleDirectoryReader(input_dir=files_path).load_data()
index = GPTVectorStoreIndex.from_documents(documents, service_context=service_context)
index.storage_context.persist(persist_dir=persist_dir)

OOrion Pax

It looks like maybe I can do this:

Plain Text

documents = []
    for file_path in files_path:
        documents.extend(SimpleDirectoryReader(input_dir=file_path).load_data())

load_data returns a List[Document] which means they should be appendable to a running list.

OOrion Pax

Is there any downside to doing it this way?

kktingyew

What you are doing is okay, and I dont see any problems.

LLogan M

Yea that seems fine to me!

OOrion Pax

Thanks @ktingyew and @Logan M

Add a reply