Hi everyone! I've been recently diving

Setting a good similarity threshold can be difficult and can lead to no results populating your prompt from your knowledge base. It's typically just better to fetch the most similar results (with some exceptions), especially because the models like gpt-4 can pretty well distinguish that the context is irrelevant and not use that to formulate a response. Also sometimes a text chunk might be mostly irrelevant but contain one small piece of data you'd still like the model to have.

The recommended size will differ quite a lot based on how you're structuring your pipeline but in my experience it does take a very large amount of data to notice a reduction in performance and these can typically be negated. Semantic search does pretty well even with a lot of data but sometimes you might find some pollution if you have a massive amount of data that is semantically similar. These cases are a good place generally to implement those similarity cutoffs.

Thanks a lot @Teemu, very informative