The community member is asking whether it is necessary to create Documents if their preprocessing pipeline outputs chunks, and whether they can just create Nodes and insert them into the index, or create a Document and put Nodes inside it.
The comments indicate that the community member can customize Documents, and that they can also create the Nodes themselves and insert them. The comments suggest that the community member should ensure the Nodes are short enough to fit into the LLM.
Is it necessary to create Documents if my preprocessing pipeline outputs chunks? ie I have some unique data type that is input into my preprocessing pipeline, that pipeline outputs chunks of each data sample with associated metadata for each chunk. Can I just create Nodes and insert those nodes into my index? Better yet, can I create a Document and put nodes inside of it?
Also in addition, you can also create the nodes yourself and insert. You'll get better results if you make sure the nodes are short enough to fit into the LLM though
Plain Text
from llama_index.schema import TextNode
node = TextNode(text="..")
