Log in
Log into community
Find answers from the community
View all posts
Related posts
Did this answer your question?
π
π
π
Powered by
Hall
Inactive
Updated last year
0
Follow
hey is there any additional information
hey is there any additional information
Inactive
0
Follow
n
niid
last year
Β·
hey is there any additional information on
https://docs.llamaindex.ai/en/stable/api/llama_index.node_parser.MetadataAwareTextSplitter.html
? What is it indended for? How does it work?
L
n
7 comments
Share
Open in Discord
L
Logan M
last year
It's a base class that is meant to be extended. The
SentenceSplitter
and
TokenTextSplitter
are both subclasses of this
L
Logan M
last year
Since metadata is included when sending text to the LLM, the text needs to be split with that metadata considered
L
Logan M
last year
That class makes it a little easier when implementing new text splitters
n
niid
last year
I am wondering in particular about SentenceSplitter.. is it used when sending text to the LLM?
n
niid
last year
I only know it from using it in IngestionPipeline
n
niid
last year
ah ok now I get. When the initial text is being split the "would-be" length of the metadata is included. So when sending to LLM in response synthesizer, the len(chunk) + len(metadata) <= chunk_size
L
Logan M
last year
you got it!
Add a reply
Sign up and join the conversation on Discord
Join on Discord