Find answers from the community

Updated 3 months ago

Does someone know how much the cost

Does someone know how much the cost differs using topK 25 vs 4

and how much the results differ
T
A
L
6 comments
That will depend on your data, mainly the chunk size. The largest factor here will be the LLM, with too much context the answer synthesis won't be as good (lost in the middle problem etc.)

Also for the cost, it mainly comes from the extra cost of passing more results to the LLM. Embeddings models are pretty cheap in comparison.

But with those numbers (assuming similar chunk sizes) it would be roughly 6x the cost.
So what are the settings you recommend
It really depends on your data.

In most cases, a higher top k with a reranking step to reduce/filter is a good choice
So how could that look if u dont mind me asking
Like is there a vid I can reference
Add a reply
Sign up and join the conversation on Discord