Find answers from the community

Updated 2 years ago

I have a question because I would like

At a glance

I have a question because I would like to do a test implementation using embedding myself.

When indexing, does llamaindex chunk the text data and throw a request to openai's embedding endpoint for the number of chunks?

3 comments

LLogan M

yea, it does batch embedding calls

By default, the batch size is 10

TTomPenguin

Thank you so much! Super engineer!

I would like to refer to the logic of chunking the text, can you please share a link to your implementation?

LLogan M

The default text splitter is our sentence splitter, which tries to split while respecting sentence boundaries

Here's the code
https://github.com/jerryjliu/llama_index/blob/9fa866ee883b37c6f1d7c6d9f7923dbe8929f4d2/llama_index/text_splitter/sentence_splitter.py#L124

Add a reply