and i would like one query engine for

and i would like one query engine for the parent folder

13 comments

You'll need to load the data probably using something like documents = SimpleDirectoryReader("./data/paul_graham").load_data()

bbmax

Plain Text

documents = SimpleDirectoryReader("./data/paul_graham").load_data()

nodes = node_parser.get_nodes_from_documents(documents)
# initialize storage context (by default it's in-memory)
storage_context = StorageContext.from_defaults()
storage_context.docstore.add_documents(nodes)

bbmax

more info on SimpledirectoryReader https://docs.llamaindex.ai/en/stable/module_guides/loading/simpledirectoryreader.html

SSenes / Samil

will try it out

bbmax

If you're trying to load the the indexes from storage and merge them into one query engine

bbmax

you'll need to use like a sub question query engine or a router

bbmax

https://docs.llamaindex.ai/en/stable/examples/query_engine/sub_question_query_engine.html

SSenes / Samil

the simpledirectoryreader would probably do the job i think

bbmax

https://docs.llamaindex.ai/en/stable/module_guides/querying/router/root.html#routers

SSenes / Samil

i'm trying to use this code in a foreach the code looks a bit like this

Plain Text

subfolders = [ f.path for f in os.scandir(path_docs) if f.is_dir() ]
            for x in subfolders:
                arr = x.split('\\')
                folder = arr[len(arr) - 1]
                reader = llama_index.SimpleDirectoryReader(input_dir=os.path.join(path_docs,folder))
                documents = reader.load_data()
                parser = SentenceSplitter()
                nodes = parser.get_nodes_from_documents(documents)
                # initialize storage context (by default it's in-memory)
                storage_context = StorageContext.from_defaults(persist_dir= path_docs)
                storage_context.docstore.add_documents(nodes)

            index = llama_index.load_index_from_storage(storage_context=storage_context)
            indexed = True

and you can guess it only works for the last document since the storage_context is reset every loop

SSenes / Samil

Also tried this :

Attachment

SSenes / Samil

but it keeps saying it requires a persist dir, and when i specify the parent dir it says no file found

SSenes / Samil

more specifically : ValueError: No index in storage context, check if you specified the right persist_dir

Add a reply

Find answers from the community

and i would like one query engine for