Find answers from the community

Updated 3 months ago

improve codesplitting

Any tips on improving the codesplitter?
s
4 comments
Like tree sitter queries can help to a certain extent, but we need rebuilding logic, metadata tagging etc. Should those be part of callbacks?
take this notebook for instance. Here I split a given java code and rebuilt based on the chunks I wanna filter out (can make nested queries too).
Issue is different ppl may have different use cases. So should we make the codesplitter as modular as possible in turn making it more difficult to use.

Or do we try to add a new package providing code spitting for language specific use cases.
Also filters on case by case basic is required. Viz. some code bases have license info at the top for each file which is not desired information to index.
Add a reply
Sign up and join the conversation on Discord