Find answers from the community

Updated 4 months ago

Customized Documents

At a glance
Is it necessary to create Documents if my preprocessing pipeline outputs chunks? ie I have some unique data type that is input into my preprocessing pipeline, that pipeline outputs chunks of each data sample with associated metadata for each chunk. Can I just create Nodes and insert those nodes into my index? Better yet, can I create a Document and put nodes inside of it?
W
L
2 comments
Also in addition, you can also create the nodes yourself and insert. You'll get better results if you make sure the nodes are short enough to fit into the LLM though

Plain Text
from llama_index.schema import TextNode

node = TextNode(text="..")
Add a reply
Sign up and join the conversation on Discord