We're indexing large data sets that we save as text files. The one in question is a set of 1600 product reviews that are collected from various product review sites. The issue seems to be that if I put them all in one index (which we store for later recall), and ask for a summary of the reviews, it only seems to consider a few of the reviews in the analysis. In this case there are 229 doc IDs in the docstore file of the index. All the docs here contain similar information of approximately 7 reviews. Is this because when I query the index and llama index sees relevant information for my query response in the first couple docs, it just stops there?
As a follow up, are there settings in LlamaIndex I should use to ensure that all docs are reviewed for relevancy prior to responding? Thoughts and help much appreciated!