Large index

LLawSnap

Noob question about working with large (230 mb) index? Hi, I used GPTSimpleVectorIndex to create an index of several hundred pdf files. When I try to query the index, I get answers but they are kind of inconsistent -- makes me think(?) that query is only grabbing some information each time but then running out of memory?

Right now I'm using the index.query as described in the "getting started" tutorial. Very new at this and so would appreciate pointers?

I reviewed recent videos on youtube that suggested

using pinecone to increase memory size (but does that improve processing?)
video that suggested progressive summarization (query applies to one chunk at time, then all answers are concatenated together and then fed back into GPT, which provides summary of the concatenation of all the answers)
video that suggests using langchain.

Any pointers appreciated. Thanks!

6 comments

LLogan M

Are you using just a single vector index? You might get better results if you set similarity_top_k=3 or similar during the query call. The default is 1.

In addition, you can try setting response_mode="compact" in the query to reduce LLM calls and improve response times with a higher top k

Using pinecone will help with memory/speed, but it shouldn't change the quality of response much, if at all

LLawSnap

Ok, thanks I'll give that a try.

LLogan M

You can also set a similarity cutoff during the query, to remove nodes below a certain threshold. Sometimes this is helpful if you increase the top_k

index.query(..., similarity_cutoff=0.5)

LLawSnap

Thanks Logan, based on the info that Jerry posted today showing all the great work y'all are doing, I think I'm going to try using pinecone and storing each document as its own node, and then using pinecone_index.query to identify the relevant documents.

LLawSnap

I also need to review all the new documentation that you all just added!

LLogan M

haha lots of new stuff to see! That sounds good though! 👍

Add a reply

Find answers from the community

Large index