Find answers from the community

Updated 2 months ago

In VectorStoreIndex.from_documents, is

In VectorStoreIndex.from_documents, is it possible to parse documents faster. The "parsing documents into nodes" step is taking sometime - hopefully it can be done in parallel if possible
L
1 comment
You can parallelize and do the parsing yourself ahead of time actually

Parsing nodes into documents is just
Plain Text
from llama_index.node_parser import SimpleNodeParser

node_parser = SimpleNodeParser.from_defaults(chunk_size=1024)

nodes = node_parser.get_nodes_from_documents(documents)


Then once you have the nodes, you can use them directly

Plain Text
index = VectorStoreIndex(nodes, ...)
Add a reply
Sign up and join the conversation on Discord