is there any way i can remove chunks that are too large automatically? I keep getting the following error:
ValueError: A single term is larger than the allowed chunk size.
Term size: 4900
Chunk size: 3900Effective chunk size: 3900
Or, if anyone knows how to split a huge text file more efficiently. I have tried using the CharacterTextSplitter, using the "." as a separator, but I fear I am doing something wrong as it looks like it is still not splitting it enough
