Find answers from the community

Updated 3 months ago

Hello I m trying to develop an app with

Hello I'm trying to develop an app with RAG pipeline with two parts:
1) Knowledge base builder - intakes documents from a storage andextracts embeddings from the documents.
2) Client part - takes user's question, returns list of the documents )

docstore = SimpleDocumentStore() docstore.add_documents(documents) vector_store = MilvusVectorStore(**config.get("vector_store")) index_store = SimpleIndexStore() storage_context = StorageContext.from_defaults(vector_store=vector_store, index_store=index_store, docstore=docstore) embed_model = OpenAIEmbedding() if os.environ.get("OPENAI_API_KEY") else "local" service_context = ServiceContext.from_defaults(embed_model=embed_model) index = VectorStoreIndex.from_documents(documents=documents, storage_context=storage_context, service_context=service_context, show_progress=True) index.storage_context.persist(config.get("storage_context")) docstore.persist(config.get("documents_store"))
Questions:
1) Do I need to parse my documents into nodes and add both docs and nodes to the docstore?
2.a) Does VectorStoreIndex.from_documents() parse nodes by default?
2.b) Can I force it to use documents?
2.c) Am I right that the only way to keep relations between nodes and documents is to parse them explicitly?
3) Do I need to pass documents to VectorStoreIndex even if storage_context is aware of docstore?
4) Does docstore update documents if I add documents with the same id, but diffennt contents?
5) Am I right that node_id is used to indentify vectors in my MilvusVectorStore?
6) Why do I need to persist storage_context if client part loads index using VectorStoreIndex.from_vector_store() ? What is correct way to save index and use it for quering?
W
L
M
4 comments
  1. You don't have to parse he documents on your own if you don't want to. You can simply pass the documents during Index creation and it will create nodes and add the nodes inside the docstore on its own.
For ref: https://gpt-index.readthedocs.io/en/stable/core_modules/data_modules/node_parsers/usage_pattern.html

2.a ) Yes
2.b) No
2.c) When you pass document at the time of index creation, Each document when being converted into nodes gets a property called include_prev_next_rel which keeps related nodes in sync.

  1. Yes, If you have added new information in the docstore, The it is required to create embeddings for the same as well.
  1. Yes docstore will update the content if you update the existing doc, You can find more here: https://gpt-index.readthedocs.io/en/latest/core_modules/data_modules/index/document_management.html#update
For 5 and 6 I'm hoping @Logan M πŸ¦‡ can pitch in.
  1. Yes
  1. You don't need to use persist with mivus, the docstore and index store are not used, unless you are setting store_nodes_override=True
@Logan M and @WhiteFang_Jr thank you for answers. What is the key concept of VectorStoreIndex? I've read the docmentation but still can't get what this object is for.
Vector store index or be it any index, all of them serve similar purpose. The serving part can different index to index but core remains the same.


VectorStoreIndex stores nodes and embeddings to itself of the given documents.
You can perform insertion/updation/deletion operations on the nodes using the index.
Index also let you create query_engine instance which helps you to query your own data.
Add a reply
Sign up and join the conversation on Discord