hi, jerry. I saw the feature on new Sentence Text splitter. it will be called automatically during the opration of creating new index? an other question is: if it can split words in languages not using white space between words, like Chinese? I am using 0.4.32 mainly, and I saw error message about over length term (longer than max_chunk_limit), so I have to process document by a chinese word splitter before creating index, thus I think the build-in splitter not fits languages without white space...
thanks for your reply. to run a specific word splitter (for me it's a Buddhism dict add on) before creating index will gain benefit at semantic understanding or not? if there is no over length single term error, and there is no benefit on semantic understanding, I won't do this before creating index. because it's a bit complicated and get some increases on token number.
a silly question maybe: to run a specific word splitter (for me it's a Buddhism dict add on) before creating index will gain benefit at semantic understanding for llm or not?
