Pinecone query

SSiddhant Saurabh

known this, you can see I have used in implementation 3
but index = GPTPineconeIndex([], pinecone_index=index)
can not be queried because it is empty, right?

4 comments

LLogan M

You can still query, the documents should be stored in the pinecone index right?

kkartik9257

yes it should be stored there

SSiddhant Saurabh

the above works. but

by default, pine cone does not store text
I went through the implementation of pinecone add and query function in gpt-index, I see you are storing text in the metadata. (neat trick applied there for doing away from the manual effort of managing mapping for text) and on pinecone query you are fetching back the text of the top_k for forming theVectorStoreQueryResult. got this much.

But I have a few concerns

in pinecone documentation it says we can have metadata upto 5 or 10kb (https://community.pinecone.io/t/metadata-size-limit-10kb-or-40kb/900)
With more heavy metadata will there be any impact on search?

LLogan M

Since we are breaking the documents into chunks (default size of 3900 tokens), I don't think the Metadata size limit should be exceeded 🤔

I don't think there is any impact on search. We store the emebdding vector, and then we query the matching ones. Maybe some slightly slower api calls due to sending the text too, but I doubt it's a big impact

Add a reply

Find answers from the community

Pinecone query