Find answers from the community

Updated 2 months ago

Index.insert() method document splitting parameters

Hey there, in index.insert() method there is parameter of type Document but I don't know how it is splitting my new document. If I want to manually define chunk_size and chunk_overlap for new documents inserted then how to do it? Or will it just take those parameters directly from already defined index?
W
$
3 comments
Yes the index.insert() takes in a Document object and chunk it based on the defined set of values.

If you want to create certain object irrespective of the defined chun_size, I would suggest you create the TextNode object which is the Parent of Document object.

Plain Text
from llama_index.core.schema import TextNode

node = TextNode(text="<text_chunk>", id_="<node_id>")

# Add this node in your index
index.insert_nodes([node])
@WhiteFang_Jr Defined set of values means globally defined or defined in index already there?
Yes globally defined, you can access them using Settings.
Plain Text
from llama_index.core import Settings

Settings.llm # Your Globally defined llm
Settings.chunk_size # defined chunk size

You can read more about this here: https://docs.llamaindex.ai/en/stable/module_guides/supporting_modules/settings/#node-parser-text-splitter
Add a reply
Sign up and join the conversation on Discord