Find answers from the community

Updated 3 months ago

In VectorStoreIndex.from_documents, is

In VectorStoreIndex.from_documents, is it possible to parse documents faster. The "parsing documents into nodes" step is taking sometime - hopefully it can be done in parallel if possible

1 comment

LLogan M

You can parallelize and do the parsing yourself ahead of time actually

Parsing nodes into documents is just

Plain Text

from llama_index.node_parser import SimpleNodeParser

node_parser = SimpleNodeParser.from_defaults(chunk_size=1024)

nodes = node_parser.get_nodes_from_documents(documents)

Then once you have the nodes, you can use them directly

Plain Text

index = VectorStoreIndex(nodes, ...)

Add a reply