Pinecone embedding

aandrewv

Hey all I am trying to use a non default embedding(trying to use hugging face) with a pinecone index but I keep on getting an error that it does not access an embed param? Here is my code for reference:

gpt_index = pinecone.Index("testing")
embed_model = LangchainEmbedding(HuggingFaceEmbeddings())
index = GPTPineconeIndex(pinecone_index=gpt_index, embedding_model=embed_model)

I am also on llama-index 0.5.8

Would appreciate any help!

27 comments

LLogan M

You'll want to pass in the embed model in the service context object

Plain Text

from llama_index import ServiceContext

index = GPTPineconeIndex(...., service_context=ServiceContext.from_defaults(embed_model=embed_model))

aandrewv

Thanks!

aandrewv

Ill try it out

aandrewv

Also is there a way to load documents from pinecone and use it as vectorstore at the same time. Main use-case I am trying to do is that I will just load all my documents from pinecone, but I am stuck on generating the query vector and the id_to_text_map, but creating index requires documents.

LLogan M

You can pass an empty array if the documents are already on pinecone

GPTPineconeIndex([], ...)

aandrewv

Oh awesome!

aandrewv

ty ty

aandrewv

Is something like this enough: embed_model =
index = GPTPineconeIndex.from_documents([], pinecone_index=gpt_index)
result = index.query("what is life?")

aandrewv

Oh shoot i get an error like KeyError: 'doc_id'

LLogan M

Is gpt_index equal to the pinecone client index? 🤔

aandrewv

Full code:

api_key = os.getenv('PTOKEN')
pinecone_environment = os.getenv('PCLOUD')

pinecone.init(api_key=api_key, environment=pinecone_environment)

gpt_index = pinecone.Index("gptindextwo")
embed_model = LangchainEmbedding(HuggingFaceEmbeddings())
index = GPTPineconeIndex([], pinecone_index=gpt_index,)
result = index.query("what is life?")

aandrewv

i don't use hugging face here, but forgot to delete it

LLogan M

Gotcha. And what's the full error/stack trace?

aandrewv

happens in query result

aandrewv

query_result = self._vector_store.query(

aandrewv

python3.10/site-packages/llama_index/vector_stores/pinecone.py"

aandrewv

id = match.metadata["id"]
KeyError: 'id'

LLogan M

Interesting 🤔 hmm I'll check something

LLogan M

I'm assuming you already created the index on pinecone? 😅https://github.com/jerryjliu/llama_index/blob/main/examples/vector_indices/PineconeIndexDemo.ipynb

aandrewv

Yeah

aandrewv

unless maybe my env are incorrect

aandrewv

thanks for helping me!

aandrewv

nope just checked 😦

LLogan M

Yea no worries! Maybe try re-creating the pinecone index? Not sure whats up here haha. Judging from the code, the Metadata from pinecone isn't what we expected https://github.com/jerryjliu/llama_index/blob/main/gpt_index/vector_stores/pinecone.py#L202

aandrewv

good idea

aandrewv

I'll give it a go

LLogan M

I'm thinking maybe the documents have to be originally inserted with llama index? 🤔 then once they are there you can load the index with that empty array.

not totally sure though, I haven't used pinecone yet lol just trying to understand the code 😅

Add a reply

Find answers from the community

Pinecone embedding