Is there any reason for chunk sizes to

programmers like powers of 2 🙂

(And generally, the max size for embeddings or llms is some power of 2)

but you can definitely make it whatever you want tbh

Does this have anything to do with effective data storage?

Like say 512 is preffered over 500?

@kush2861 Good question.

I think you’re correct in assuming that it is because of how data is stored (& therefore retrieved i.e. memory alignment) .

We can use any chunk size though. But I imagine that power of 2s might be more optimal on average

Thanks for the response @LunarShadow