The community member is building a QA bot for accounting and law-related questions, and is using a vector index to store embeddings of the text data. They are facing an issue where a high similarity_top_k of 8 is required to reliably select relevant nodes, but this leads to multiple OpenAI API calls, which they would like to limit. The community members discuss potential solutions, such as experimenting with smaller chunk sizes or using a different index structure like a knowledge graph. However, there is no explicitly marked answer in the comments.
is there a way to set the max number of refinement rounds with response_mode = "compact"? because of the nature of my data, I need to go with similarity_top_k = 8 and sometimes, it yields too many open ai requests - would like to limit it
ATM my chunk_size is 1024. my use case is that I am building a QA bot capable to answer questions related to accounting and laws related to that in my homecountry. data for that is basically a bunch of laws - each law is a hierarchical piece of text structured into multiple articles, sub-articles etc.
my current approach is that I am using a vector index to store embeddings for each law (enhancing it with metadata about article number etc), however, the phrasings of the laws is oftentimes very similar and I am running into a problem that a top_k of 2 is not enough to reliably select a relevant nodes.
what I've found is that top_k of 8 works reliably enough, but requires at least two open ai API calls (one initial and one refinement). sadly, I wasn't able to figure out a better alternative to this, maybe using a totally different index might work better in my case
thank you - will consider PR, you've built an awesome library. if you have any pointers about how to improve the index structure for my use case, would appreciate that tremendously, but not sure if questions like that are in scope of this channel 😄
hmm interesting problem! That means the the vectors have trouble figuring out stuff. Not sure on a better approach just yet, but maybe making the chunk size smaller (or larger, but then decreasing the top_k) will help?
