can someone guide me how to connect

iisra_j

can someone guide me how to connect Llama index with pinecone, im unable to find any code in Javascript all the code available is in python.

18 comments

RRohan

Plain Text

import {
  PineconeVectorStore,
  SimpleDirectoryReader,
  storageContextFromDefaults,
  VectorStoreIndex,
} from "llamaindex";


const reader = new SimpleDirectoryReader();
const documents = await reader.loadData({ directoryPath: sourceDir });

const pineconeVectorStore = new PineconeVectorStore();

const storageContext = await storageContextFromDefaults({ vectorStore: pineconeVectorStore });

const index = await VectorStoreIndex.fromDocuments(documents, {
      storageContext: storageContext,
    });

const queryEngine = await index.asQueryEngine();

Details: https://github.com/run-llama/LlamaIndexTS/tree/f90f7fee641b36aa7b5a61d88d87e508921b1849/examples/pinecone-vector-store

iisra_j

Kindly confirm the work flow, first the index will be created on pinecone, then have to connect the pinecone with Llama index. Then create embedding of the documents and upsert that embedding in the pinecone created index ?

RRohan

Yeah, but the index will not be automatically created.

I forgot to mention the environment variables in the last message.
PINECONE_API_KEY
PINECONE_ENVIRONMENT
PINECONE_INDEX_NAME (default "llama")

You'll have to create the index you're gonna use in environment variable beforehand.

Attachment

iisra_j

index will be created like this right ?

const client = new Pinecone({
            apiKey: process.env.PINECONE_API_KEY
        })
await client.createIndex({
            name: "llama,
            dimension: vectorDimension,
            metric: 'cosine',
            spec: {
                pod: {
                    environment: 'gcp-starter',
                    podType: 'p1.x1'
                }
            }
        });

RRohan

yep

iisra_j

the index name won't be passed anywhere in PineconeVectorStore instance ? I saw the Python code in the documentation like this
vector_store = PineconeVectorStore(pinecone_index=pinecone_index)

RRohan

it'll use the value of PINECONE_INDEX_NAME environment variable (or 'llama' by default)

You can also pass the index name like you mentioned above:

Plain Text

const pineconeVectorStore = new PineconeVectorStore({
  indexName: "myIndex",
});

This will get precedence over the environment variable

iisra_j

@Rohan can you tell before upserting the embeddings in the pinecone, how can i add extra info in the metadata. using javascript. please help

RRohan

after you get the documents, you can add your custom metadata to the documents like this:

Plain Text

documents.forEach(document => {
  document.metadata = {
    meta1: "value1",
    meta2: "value2"
    ...
  }
})

iisra_j

Thank you for replying.

iisra_j

@Rohan can you please help me. I am trying to run the query but im getting this error
error: The argument to query accepts multiple types. Either 1) must not have property: vector. Must have required property: id. There were also validation errors: argument must NOT have additional properties, argument must NOT have additional properties. 2) had validation errors: argument must NOT have additional properties, argument must NOT have additional properties.

Here is my code :

try {
    let question = "Who is trespasser ?";
    const pineconeClient = new PineconeVectorStore();
    const serviceContext = serviceContextFromDefaults();
    const index = await VectorStoreIndex.fromVectorStore(pineconeClient, serviceContext);
    // Query the index
    const queryEngine = await index.asQueryEngine();
    try {
      const answer = await queryEngine.query({ query: question, stream: true })
      //console.log("🚀🚀 Response == ",answer.response);
    } catch (error) {
      console.log("🚀 ~ exports.LlamaQueryResponse=onCall ~ error:", error.message)
      // new HttpsError("Error inner", error);
    }
  } catch (error) {
    console.log("🚀 ~ exports.LlamaQueryResponse=onCall ~ error:", error)
    logger.error(error);
    // new HttpsError("Error", error);
  }

RRohan

unless I'm missing something, looks like the code should work. What's the exact error message you're getting?

iisra_j

Attachment

iisra_j

Following are the version of pinecone and llama index
"@pinecone-database/pinecone": "^2.0.1",
"llamaindex": "^0.1.3",

iisra_j

Even when i try to run the pinecone example im getting the same error when querying
The example im talking about is
https://github.com/run-llama/LlamaIndexTS/tree/f90f7fee641b36aa7b5a61d88d87e508921b1849/examples/pinecone-vector-store

ootpi

@Rohan is there a sample you can point me to where we have docStore setup like you do in the above code. I am kind a confused as to how to setup a Docstore. Appreciate your help!
const storageContext = await storageContextFromDefaults({ vectorStore: pineconeVectorStore });

RRohan

something like this:

Plain Text

import { SimpleDocumentStore } from "llamaindex"

const storageContext = await storageContextFromDefaults({
   vectorStore: pineconeVectorStore,
   docStore: new SimpleDocumentStore();
});

ootpi

Hey @Rohan - My scenario is to store index in Redis/pinecone and also store the index id somewhere. Later when there is query on the documents I indexed, I should be able to create an index out of the indexId. Please see the python code here
https://discord.com/channels/1059199217496772688/1133167189860565033/1232101699292893194

My understanding of LlamaIndexTS is:

Doesn't support Redis (as tested ok in my py implementation)
Only supports SimpleDocumentStore, so it can only save to local disk

So for scenarios, where there is a lot of documents, where I cannot save locally in a cloud/server environment, I cannot have a DocStore and separately save Vector and SummaryIndex which I can load at a later time.

The following piece of code if can be done in TS will be very helpful: 🙏 Redis is not important but storing and loading like this is.

storage_context = StorageContext.from_defaults(
    docstore=RedisDocumentStore.from_host_and_port(host=REDIS, port=19282, namespace="llama_index"),
    index_store=RedisIndexStore.from_host_and_port(host=REDIS, port=19282, namespace="llama_index"),
)

storage_context.docstore.add_documents(nodes)
summary_index = SummaryIndex(nodes, storage_context=storage_context)
vector_index = VectorStoreIndex(nodes, storage_context=storage_context)
keyword_table_index = SimpleKeywordTableIndex(
    nodes, storage_context=storage_context
)

# note down index IDs
list_id = summary_index.index_id
vector_id = vector_index.index_id
keyword_id = keyword_table_index.index_id

Then later I can do:
# load indices
summary_index = load_index_from_storage(
    storage_context=storage_context, index_id=list_id
)
vector_index = load_index_from_storage(
    storage_context=storage_context, index_id=vector_id
)
keyword_table_index = load_index_from_storage(
    storage_context=storage_context, index_id=keyword_id
)

Really appreciate your time. Thanks

Add a reply

Find answers from the community

can someone guide me how to connect