I am working with llama index to convert a sql database to pinecone vector database. How do I ensure that when am adding the row data to index , I drop duplicates for rows already in the vector store? The Sql database is frequently updated.
lets say I have a vector_store_index of keywords like (goat, sheep, cow, iron, zinc, copper, iron sheets etc), how do I use Llama index to return the main keywords related to say a new word like 'donkey' and their similarity score using cosine?