Hi, I am having a problem with using lancedb as vectore store. My documents include metadata, but the metadata is lost after I construct the vectorestore index. When I persist the storage context, the
docstore.json is empty. This problem does not exist when I use simple vector store or FAISS vector store. Could you please let me know what am I doing wrong from the following codes?
How I create the document:
doc = Document(
text=record['text'],
extra_info= {k: v for k, v in record.items() if k in extra_info_fields}
)How I create the index:
vector_store = LanceDBVectorStore(uri="lancedb_storage")
lancedb_storage_context = StorageContext.from_defaults(vector_store=vector_store)
lancedb_vector_retriever_index = VectorStoreIndex(
input_documents,
storage_context=lancedb_storage_context
)How I persist the context:
lancedb_storage_context.persist(persist_dir="lancedb_storage")
Sample retrieved node:
NodeWithScore(node=TextNode(id_='3660f5fe-fca9-4244-9b04-d64685aa796f', embedding=None, metadata={}, excluded_embed_metadata_keys=[], excluded_llm_metadata_keys=[], relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='None', node_type=None, metadata={}, hash=None)}, hash='b60d4828f419a879a0ec2619aaf35fa5b677a1c5b67c6b59bc28123ddf04641f', text="...", start_char_idx=None, end_char_idx=None, text_template='{metadata_str}\n\n{content}', metadata_template='{key}: {value}', metadata_seperator='\n'), score=0.3688541352748871)