LlamaIndex
Log in

Find answers from the community

s
F
Y
a
P
Updated 2 years ago

Text splitting

RRay Li
·
awesome! I was wondering if there are Any best practices for choosing the appropriate node length? IMHO a larger window may adds too much noise yet a small window might lose context info/
L
R
5 comments
LLogan M
My best advice is to try and group things into logical groups
LLogan M
Like, if you have clear sections that you can parse out, those make good document splits
LLogan M
Otherwise, overlapping chunks usually works ok-ish
RRay Li
I see. I guess I can use Langchain's text splitter to build nodes and use Llama index, since they offer a few different ones.
LLogan M
Yea for sure!
Add a reply
Sign up and join the conversation on Discord