Find answers from the community

Updated 3 months ago

Hi I am trying to customize the

Hi, I am trying to customize the SentenceWindowNodeParser e.g. change the chunk_length and the splitter. How can I do it?

7 comments

bbmax

Check this out: https://gpt-index.readthedocs.io/en/stable/core_modules/data_modules/node_parsers/usage_pattern.html#customization

bbmax

If you're using the SentenceWindowNodeParser it doesn't look like you can customize the splitter,

you can only customize sentence_splitter etc

If you want more customization try the SimpleNodeParser

LLogan M

You can actually change the splitter, but it's a bit janky (so I decided not to expose it in the docs lol)

In from_defaults(), you can pass in sentence_splitter which is a callable function with str in , List[str] out

PPujari12

@Logan M Hi, I have tried doing it but getting some errors. Can you give me an example of how to pass the sentence_splitter into it?

bbmax

https://github.com/jerryjliu/llama_index/blob/main/llama_index/text_splitter/utils.py#L33 this is the default one that's sent in

LLogan M

Here's a very bad/naive example

Plain Text

def my_splitter(text: str) -> List[str]:
  return text.split('. ')

node_parser = SentenceWindowNodeParser.from_defaults(..., sentence_splitter=my_splitter)

PPujari12

got it thanks

Add a reply