Hello, Is there a way to determine when to stop a chunk ? I want to have chunks that stops at the end of a paragraph and does not overlap. I’ve tried with sentence splitter with \n\n\n but it does not seems to be doing it. The other option would be to have hundreds of separated little txt files but I’d rather not
@Logan M hello, I’ve forked the repo and modified the dented splitter class to have the behavior I want, which is to have chunks either of size x (256,512…) or have a chunk of size smaller than d that stops at a new paragraph. Do you think this is a useful feature for other person ? Like having a end_chunk_separator parameter or would this PR be useless ?