----------------------
Yes, you can specify separators for
TokenTextSplitter. In the provided extracts, you can see an example of specifying a separator when creating a
TokenTextSplitter instance:
from gpt_index.langchain_helpers.text_splitter import TokenTextSplitter
text_splitter = TokenTextSplitter(separator=" ", chunk_size=2048, chunk_overlap=20)
In this example, the separator is set to a single space
" ". You can change the
separator parameter to any string you'd like to use as a separator.
Source:
GitHub - InsertDemo.ipynb (
https://github.com/jerryjliu/gpt_index/tree/main/examples/paul_graham_essay/InsertDemo.ipynb)
----------------------
Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:
----------------------
