Find answers from the community

Updated 8 months ago

I was able to find in the docs a few

I was able to find in the docs a few months ago but i can't find it anymore.
How can I use text_splitter = SentenceSplitter() and split sentences by Asian Chinese/Japanese characters?

5 comments

LLogan M

secondary_chunking_regex="[^,.;。？！]+[,.;。？！]?"

LLogan M

Probably modifying that?

LLogan M

Thats the default

cchantlong

oh the default already includes the Japanese characters like 。

cchantlong

thanks

Add a reply