Find answers from the community

Updated 3 months ago

Pinecone query

known this, you can see I have used in implementation 3
but index = GPTPineconeIndex([], pinecone_index=index)
can not be queried because it is empty, right?
L
k
S
4 comments
You can still query, the documents should be stored in the pinecone index right?
yes it should be stored there
the above works. but

by default, pine cone does not store text
I went through the implementation of pinecone add and query function in gpt-index, I see you are storing text in the metadata. (neat trick applied there for doing away from the manual effort of managing mapping for text) and on pinecone query you are fetching back the text of the top_k for forming theVectorStoreQueryResult. got this much.

But I have a few concerns
  1. in pinecone documentation it says we can have metadata upto 5 or 10kb (https://community.pinecone.io/t/metadata-size-limit-10kb-or-40kb/900)
  2. With more heavy metadata will there be any impact on search?
  1. Since we are breaking the documents into chunks (default size of 3900 tokens), I don't think the Metadata size limit should be exceeded πŸ€”
  1. I don't think there is any impact on search. We store the emebdding vector, and then we query the matching ones. Maybe some slightly slower api calls due to sending the text too, but I doubt it's a big impact
Add a reply
Sign up and join the conversation on Discord