Hey I want to index a python codebase

Solitary · 2023-09-12T08:25:04.350Z

Hey I want to index a python codebase and the default Textsplitter is SentenceSplitter. How do I change this to do code aware chunking?

You can choose CodeSplitter like this to change default settings

Plain Text

from llama_index.text_splitter import CodeSplitter
import tiktoken

text_splitter = CodeSplitter(
  separator=" ",
  chunk_size=1024,
  chunk_overlap=20,
  tokenizer=tiktoken.encoding_for_model("gpt-3.5-turbo").encode
)

node_parser = SimpleNodeParser.from_defaults(text_splitter=text_splitter)

Define the parameters as per your requirements. For more You can refer here
https://github.com/jerryjliu/llama_index/blob/main/llama_index/text_splitter/code_splitter.py

Find answers from the community

Hey I want to index a python codebase