Find answers from the community

Updated 2 months ago

like a foreach of a parent folder, and

like a foreach of a parent folder, and loading each child folder that contain their docstores
L
S
21 comments
You can get the nodes from each docstore with

nodes = list(index.docstore.docs.values())

Then you could shove all those into a single index
It would re-embed them though, unless you did some hax to get the embeddings from the vector_store
i dont think its a problem if it re-embeds them
right now i have a way to get the nodes of each docs
Yea, so then you can gather up a giant list of nodes using the above, then do

index = VectorStoreIndex(nodes, ...)
its smth like that :
Attachment
image.png
this code only loads the last doc as you can guess
since it resets the storage context
so basically i dont use the storage context
and build the index with all the nodes
hmm. Maybe I'm confused at what you are trying to do lol

But either way, you just need to build a master list of nodes, and then give those nodes to the index constructor
i understood that part but how will you build your list ?
does it have like a build in llama_index way ? or a simple py list of nodes ?
this is how i do it when i want to load a single document from storage, but what i want to do, is load every documents within a folder from storage
Attachment
image.png
originally its what i wanna do
this shows an error stating that i gave 0 argument in the append which is weird
Attachments
image.png
image.png
the node parser returns a list
soo do we give it a list of documents ?
so you'd probably want

Plain Text
all_nodes = []
for ...:
  nodes = node_parser...
  all_nodes.extend(nodes)
index = VectorStoreIndex(all_nodes)
Thanks, this worked like a charm
Add a reply
Sign up and join the conversation on Discord