Uploading large files to a vector store with Pinecone
Uploading large files to a vector store with Pinecone
At a glance
The community member is facing issues uploading large amounts of data to a Pinecone vector store. They are encountering timeout errors when using the ingestion pipeline, and the parallel processing approach is resulting in a "Resource already exists" error (409). The community members have suggested trying to lower the batch_size parameter in the PineconeVectorStore constructor to address the timeout issue, but the community member is still concerned about how to upload large amounts of data in less time. There is no explicitly marked answer in the comments.
Hii dudes, when i upload embeddings to pinecone using ingestion pipeline(by pipeline.run) it interrupts in the middle due to the timeout error. and when i try with parrallel processing using " num_workers "option it interrupts with the error409- error":{"code":"ALREADY_EXISTS","message":"Resource already exists"},"status":409. what is the sollution to upload large amount of file to the vectorstore? @Logan M
