Text splitting

For code, I feel you don't want to split the code into chunks where lines are cut off.

The default text splitter does this. It works fine for normal text, but for code, my intuition tells me that might be causing issue, and something smarter can probably be done

You can use any text splitter from langchain. There's a list of splitters here: https://langchain.readthedocs.io/en/latest/reference/modules/text_splitter.html

Then when you construct your index, you can pass it in with text_splitter=MyTextSplitter() or something like that

Excellent! I really appreciate the answer. I share your intuition. I had the feeling from the start that code needed to be handled differently than normal text, but I haven't been able to find any app that utilizes Llamaindex for code.