Find answers from the community

Updated 2 months ago

Text splitting

is there any way i can remove chunks that are too large automatically? I keep getting the following error:

Plain Text

ValueError: A single term is larger than the allowed chunk size.
Term size: 4900
Chunk size: 3900Effective chunk size: 3900

Or, if anyone knows how to split a huge text file more efficiently. I have tried using the CharacterTextSplitter, using the "." as a separator, but I fear I am doing something wrong as it looks like it is still not splitting it enough

6 comments

LLogan M

So are you setting both a prompt helper and node parser then?

ssilenth

yes

LLogan M

Did you set the chunk size limit in both the node parser and prompt helper?

ssilenth

i believe i have in the past, but let me try that again. Previously, it hadn't worked for me. I set it pretty high, or at least a few above the Term that was causing the error

LLogan M

Try setting it around 1500 maybe?

ssilenth

will do

Add a reply