Find answers from the community

Updated 3 months ago

can someone guide me how to connect

can someone guide me how to connect Llama index with pinecone, im unable to find any code in Javascript all the code available is in python.
R
i
o
18 comments
Plain Text
import {
  PineconeVectorStore,
  SimpleDirectoryReader,
  storageContextFromDefaults,
  VectorStoreIndex,
} from "llamaindex";


const reader = new SimpleDirectoryReader();
const documents = await reader.loadData({ directoryPath: sourceDir });

const pineconeVectorStore = new PineconeVectorStore();

const storageContext = await storageContextFromDefaults({ vectorStore: pineconeVectorStore });

const index = await VectorStoreIndex.fromDocuments(documents, {
      storageContext: storageContext,
    });

const queryEngine = await index.asQueryEngine();


Details: https://github.com/run-llama/LlamaIndexTS/tree/f90f7fee641b36aa7b5a61d88d87e508921b1849/examples/pinecone-vector-store
Kindly confirm the work flow, first the index will be created on pinecone, then have to connect the pinecone with Llama index. Then create embedding of the documents and upsert that embedding in the pinecone created index ?
Yeah, but the index will not be automatically created.

I forgot to mention the environment variables in the last message.
PINECONE_API_KEY
PINECONE_ENVIRONMENT
PINECONE_INDEX_NAME (default "llama")

You'll have to create the index you're gonna use in environment variable beforehand.
Attachment
image.png
index will be created like this right ?
const client = new Pinecone({ apiKey: process.env.PINECONE_API_KEY }) await client.createIndex({ name: "llama, dimension: vectorDimension, metric: 'cosine', spec: { pod: { environment: 'gcp-starter', podType: 'p1.x1' } } });
the index name won't be passed anywhere in PineconeVectorStore instance ? I saw the Python code in the documentation like this
vector_store = PineconeVectorStore(pinecone_index=pinecone_index)
it'll use the value of PINECONE_INDEX_NAME environment variable (or 'llama' by default)

You can also pass the index name like you mentioned above:
Plain Text
const pineconeVectorStore = new PineconeVectorStore({
  indexName: "myIndex",
});


This will get precedence over the environment variable
@Rohan can you tell before upserting the embeddings in the pinecone, how can i add extra info in the metadata. using javascript. please help
after you get the documents, you can add your custom metadata to the documents like this:

Plain Text
documents.forEach(document => {
  document.metadata = {
    meta1: "value1",
    meta2: "value2"
    ...
  }
})
Thank you for replying.
@Rohan can you please help me. I am trying to run the query but im getting this error
error: The argument to query accepts multiple types. Either 1) must not have property: vector. Must have required property: id. There were also validation errors: argument must NOT have additional properties, argument must NOT have additional properties. 2) had validation errors: argument must NOT have additional properties, argument must NOT have additional properties.

Here is my code :
try { let question = "Who is trespasser ?"; const pineconeClient = new PineconeVectorStore(); const serviceContext = serviceContextFromDefaults(); const index = await VectorStoreIndex.fromVectorStore(pineconeClient, serviceContext); // Query the index const queryEngine = await index.asQueryEngine(); try { const answer = await queryEngine.query({ query: question, stream: true }) //console.log("πŸš€πŸš€ Response == ",answer.response); } catch (error) { console.log("πŸš€ ~ exports.LlamaQueryResponse=onCall ~ error:", error.message) // new HttpsError("Error inner", error); } } catch (error) { console.log("πŸš€ ~ exports.LlamaQueryResponse=onCall ~ error:", error) logger.error(error); // new HttpsError("Error", error); }
unless I'm missing something, looks like the code should work. What's the exact error message you're getting?
Following are the version of pinecone and llama index
"@pinecone-database/pinecone": "^2.0.1",
"llamaindex": "^0.1.3",
Even when i try to run the pinecone example im getting the same error when querying
The example im talking about is
https://github.com/run-llama/LlamaIndexTS/tree/f90f7fee641b36aa7b5a61d88d87e508921b1849/examples/pinecone-vector-store
@Rohan is there a sample you can point me to where we have docStore setup like you do in the above code. I am kind a confused as to how to setup a Docstore. Appreciate your help!
const storageContext = await storageContextFromDefaults({ vectorStore: pineconeVectorStore });
something like this:
Plain Text
import { SimpleDocumentStore } from "llamaindex"

const storageContext = await storageContextFromDefaults({
   vectorStore: pineconeVectorStore,
   docStore: new SimpleDocumentStore();
});
Hey @Rohan - My scenario is to store index in Redis/pinecone and also store the index id somewhere. Later when there is query on the documents I indexed, I should be able to create an index out of the indexId. Please see the python code here
https://discord.com/channels/1059199217496772688/1133167189860565033/1232101699292893194

My understanding of LlamaIndexTS is:
  1. Doesn't support Redis (as tested ok in my py implementation)
  2. Only supports SimpleDocumentStore, so it can only save to local disk
So for scenarios, where there is a lot of documents, where I cannot save locally in a cloud/server environment, I cannot have a DocStore and separately save Vector and SummaryIndex which I can load at a later time.

The following piece of code if can be done in TS will be very helpful: πŸ™ Redis is not important but storing and loading like this is.

storage_context = StorageContext.from_defaults( docstore=RedisDocumentStore.from_host_and_port(host=REDIS, port=19282, namespace="llama_index"), index_store=RedisIndexStore.from_host_and_port(host=REDIS, port=19282, namespace="llama_index"), ) storage_context.docstore.add_documents(nodes) summary_index = SummaryIndex(nodes, storage_context=storage_context) vector_index = VectorStoreIndex(nodes, storage_context=storage_context) keyword_table_index = SimpleKeywordTableIndex( nodes, storage_context=storage_context ) # note down index IDs list_id = summary_index.index_id vector_id = vector_index.index_id keyword_id = keyword_table_index.index_id Then later I can do: # load indices summary_index = load_index_from_storage( storage_context=storage_context, index_id=list_id ) vector_index = load_index_from_storage( storage_context=storage_context, index_id=vector_id ) keyword_table_index = load_index_from_storage( storage_context=storage_context, index_id=keyword_id )

Really appreciate your time. Thanks
Add a reply
Sign up and join the conversation on Discord