Find answers from the community

Updated last year

Splitters

I have exported .docx from google and I have this type of newlines, is this text splitter a good solution?

Plain Text

text_splitter = TokenTextSplitter(
  separator=" ",
  chunk_size=1024,
  chunk_overlap=20,
  backup_separators=["\n", "\n\n", "\n\n\n", "\n\n\n\n", "\n\n\n\n\n", "\n\n\n\n\n\n", "\n\n\n\n\n\n\n", "\n\n\n\n\n\n\n\n", "\n\n\n\n\n\n\n\n\n", "\n\n\n\n\n\n\n\n\n\n"]
)

Attachment

3 comments

LLogan M

I mean, that's probably fine? But also, you could pre-process the text to cut down on the number of newlines too 😅

GGianluca

thank you logan, your help is so much appreciated

GGianluca

:peepohugback:

Add a reply