Find answers from the community

Updated last year

FaissVectorStore saving/loading changes

FaissVectorStore saving/loading changes? My old code for generating/saving and loading an index no longer works.
Plain Text
def construct_index_from_nodes(nodes, persist_dir):
    load_dotenv()
    openai_api_key = os.getenv('OPENAI_API_KEY')

    # Generate Node Embeddings
    embed_model = OpenAIEmbedding(api_key=openai_api_key)

    for node in tqdm(nodes, desc="Generating Node Embeddings"):
        node_embedding = embed_model.get_text_embedding(
            node.get_content(metadata_mode="all")
        )
        node.embedding = node_embedding

    # Build the Index from the Nodes
    print("Building Index from Nodes...")
    d = 1536
    faiss_index = faiss.IndexFlatL2(d)
    vector_store = FaissVectorStore(faiss_index=faiss_index)
    storage_context = StorageContext.from_defaults(vector_store=vector_store)

    index = VectorStoreIndex(
        nodes=nodes,
        storage_context=storage_context,
    )

    # save the resulting index to disk so that we can use it later
    print("Index created. Saving to disk...")
    index.storage_context.persist(persist_dir=persist_dir)
    print("Complete.")


def test_index(persist_dir, test_query, similarity_top_k=3):
    vector_store = FaissVectorStore.from_persist_dir(persist_dir)
    storage_context = StorageContext.from_defaults(
        vector_store=vector_store, persist_dir=persist_dir)
    index = load_index_from_storage(storage_context=storage_context)
    retriever = index.as_retriever(similarity_top_k=similarity_top_k)

    nodes = retriever.retrieve(test_query)
    for i, node in enumerate(nodes):
        print("NODE", i, "[", round(node.get_score(), 2), "]", ":", node.dict()["node"]["text"], "\n\n")

Used to produce:
Plain Text
[docstore.json, graph_store.json, index_store.json, vector_store.json]

Now produces:
Plain Text
[default__vector_store.json, docstore.json, graph_store.json, image__vector_store.json, index_store.json]
L
M
8 comments
This is correct --default__vector_store.json is the same as the old file. The saving and loading should work the same as before though with the latest version of llama-index
I just tested this the other day actually
I am on the most recent update, 0.9.15. I am getting
Plain Text
ValueError: No existing llama_index.vector_stores.faiss found at ./storage\default__vector_store.json. 
Maybe reinstall llama-index or try a fresh venv? Just installed a fresh env and ran this code, and it worked

Maybe double check you are passing in a directory, and not the full path to the json too
Attachment
image.png
My directory is not ./storage it’s ./indexes/index-name/. So the fact that it has ./storage in the error makes me think that is a bug. And I’m using the same path for saving and loading
still works
Attachment
image.png
Try updating to the latest version -- 0.9.15.post2
That worked, thank you.
Add a reply
Sign up and join the conversation on Discord