TextNode
to an IndexNode
in LlamaIndex, you can use the from_text_node
class method provided by the IndexNode
class. This method takes a TextNode
and an index_id
as arguments and returns an IndexNode
.from llama_index.core.schema import TextNode, IndexNode # Create a TextNode text_node = TextNode(text="<text_chunk>", id_="<node_id>") # Convert TextNode to IndexNode index_node = IndexNode.from_text_node(node=text_node, index_id="<index_id>")
<text_chunk>
with the text content you want to store in the node, <node_id>
with the unique identifier for the node, and <index_id>
with the unique identifier for the index.TextNode
instances that you want to convert to IndexNode
instances, you can simply use a loop to process each TextNode
. Here's an example:from llama_index.core.schema import TextNode, IndexNode # Assume text_nodes is a list of TextNode instances text_nodes = [TextNode(text=f"<text_chunk_{i}>", id_=f"<node_id_{i}>") for i in range(10)] # Convert each TextNode to an IndexNode index_nodes = [IndexNode.from_text_node(node=text_node, index_id=f"<index_id_{i}>") for i, text_node in enumerate(text_nodes)]
<text_chunk_{i}>
, <node_id_{i}>
, and <index_id_{i}>
with the text content, node identifier, and index identifier for each node, respectively.VectorStoreIndex
with IndexNode
instances, you can pass the list of IndexNode
instances to the VectorStoreIndex
constructor. Here's an example:from llama_index.core import VectorStoreIndex # Assume index_nodes is a list of IndexNode instances index = VectorStoreIndex(index_nodes)
index_nodes
with your list of IndexNode
instances.StorageContext
to the VectorStoreIndex
constructor, especially if you're using a custom vector store. The StorageContext
can be created using the StorageContext.from_defaults
method and the custom vector store as an argument.from llama_index.core import VectorStoreIndex, StorageContext from llama_index.vector_store import YourCustomVectorStore # Assume index_nodes is a list of IndexNode instances # Assume your_vector_store is an instance of YourCustomVectorStore storage_context = StorageContext.from_defaults(vector_store=your_vector_store) index = VectorStoreIndex(index_nodes, storage_context=storage_context)
YourCustomVectorStore
with the class of your custom vector store, and your_vector_store
with an instance of your custom vector store.SentenceSplitter
class to split your base nodes into smaller chunks, and then create IndexNode
instances from these chunks. These IndexNode
instances will have the same node_id
as the base node, effectively creating a reference to the bigger chunk.from llama_index.core.node_parser import SentenceSplitter from llama_index.core.schema import IndexNode # Define the sizes for your sub-chunks sub_chunk_sizes = [128, 256, 512] # Create SentenceSplitter instances for each sub-chunk size sub_node_parsers = [SentenceSplitter(chunk_size=c, chunk_overlap=20) for c in sub_chunk_sizes] # Assume base_nodes is a list of your base nodes all_nodes = [] for base_node in base_nodes: for n in sub_node_parsers: # Split the base node into sub-nodes sub_nodes = n.get_nodes_from_documents([base_node]) # Create IndexNode instances from the sub-nodes sub_inodes = [IndexNode.from_text_node(sn, base_node.node_id) for sn in sub_nodes] # Add the IndexNode instances to your list all_nodes.extend(sub_inodes) # Also add the original base node to your list original_node = IndexNode.from_text_node(base_node, base_node.node_id) all_nodes.append(original_node)
base_nodes
with your list of base nodes.RecursiveRetriever
. The RecursiveRetriever
will retrieve the smaller chunks and then follow the references to retrieve the bigger chunks, providing more context for synthesis.