Find answers from the community

Updated 4 months ago

Hey How does Llama Index split long

At a glance
Hey. How does Llama Index split long texts? Eg langchain provides some features (textsplitter, recursiveTextSplitter, SpacySplitter and so on). What about Llama Index?
j
1 comment
Hi, we split the text under the hood by default, but you can choose to use a langchain text splitter too! (e.g. pass in a text_splitter arg when building an index https://gpt-index.readthedocs.io/en/latest/reference/indices.html#module-gpt_index.indices.base)

You could also choose to "pre-split" the text into separate Document objects before feeding into gpt index
Add a reply
Sign up and join the conversation on Discord