Find answers from the community

Updated last year

Is it possible to load a JSON based

At a glance

The community member asked if it is possible to load a JSON-based vector index from storage and get a list of all the nodes and their embeddings, so that the list can be added to another type of index, such as an OpenSearchVectorStore. Another community member responded that this is a bit tricky, as the embeddings are stored separately from the nodes, but provided a "slightly hacky" solution. The solution involves loading the index, getting the nodes and embeddings, attaching the embeddings to the nodes, and then creating a new index with the updated nodes. The original community member thanked the other member for the solution, indicating that it worked.

rrichie404

Is it possible to load a JSON-based vector index from storage (docstore.json, vector_store.json, etc.) and somehow get a list of all the nodes and their embeddings (call it all_nodes) so that all_nodes can be added to another type of index? Like this: vector_store.add(all_nodes_from_json) where vector_store is an OpenSearchVectorStore?

4 comments

WWhiteFang_Jr

I'm not sure, Every VectorStore Index works differently.

LLogan M

With the base vector index, since the embeddings are stored separately from the nodes, this is a little tricky

But, there is a way that is slightly hacky

LLogan M

Plain Text

# load the index
index = load_index_from_storage(...)

# get the nodes and embeddings
nodes = index.docstore.docs
embeddings = index.vector_store._data.embedding_dict

# attach the embeddings
nodes_with_embeddings = []
for node_id, node in nodes.items():
  node.embedding = embeddings[node_id]
  nodes_with_embeddings.append(node)

# create a new index with the new backend
vector_index = VectorStoreIndex(nodes_with_embeddings, storage_context=storage_context)

rrichie404

Thank you @Logan M ! That did the trick!

Add a reply