Is it necessary to create Documents if my preprocessing pipeline outputs chunks? ie I have some unique data type that is input into my preprocessing pipeline, that pipeline outputs chunks of each data sample with associated metadata for each chunk. Can I just create Nodes and insert those nodes into my index? Better yet, can I create a Document and put nodes inside of it?
Also in addition, you can also create the nodes yourself and insert. You'll get better results if you make sure the nodes are short enough to fit into the LLM though
Plain Text
from llama_index.schema import TextNode
node = TextNode(text="..")
