Find answers from the community

Home
Members
parthbs
p
parthbs
Offline, last seen last week
Joined November 28, 2024
How to create Custom Parser for attached data sample ?

I am using this snippet for text splitter and sentence splitter.

# Setup text splitter and node parser splitter = TokenTextSplitter( chunk_size=1024, chunk_overlap=256, separator=" ", backup_separators=["\n"], ) node_parser = SentenceSplitter( chunk_size=1024, chunk_overlap=256, include_prev_next_rel=True, include_metadata=True, paragraph_separator="-->" )


I want to create sentence splitter with "-->" as available in data.txt. Each node should have seperate Articles. Reason is each articles are different from each other and independent. So if anybody can help me with the configuration ?
1 comment
L