Persist dir

᲼᲼᲼᲼᲼᲼᲼᲼᲼᲼᲼᲼

Why when i do this
storage_context = StorageContext.from_defaults(
docstore=SimpleDocumentStore.from_persist_dir(persist_dir=persist_dir),
vector_store=SimpleVectorStore.from_persist_dir(persist_dir=persist_dir),
index_store=SimpleIndexStore.from_persist_dir(persist_dir=persistdir),)It tells me it cant find the docstore.json file ? Im trying to create it --

᲼

4 comments

LLogan M

I think by calling from_persist_dir, it's trying to load an existing index

When starting from scratch, from_defaults() on its own is enough

Then to persist, your just need to do storage_context.persist(persist_dir="./dir")

᲼᲼᲼᲼᲼᲼᲼᲼᲼᲼᲼᲼

Hm that should be better documented i'd say and tbf i was really thinking it would create the file itself, not really intuitive also it will add some ugly boilerplate ^^'
Thanks for the help though !

LLogan M

Yea for sure! Just to be clear, it's actually not too ugly.

Plain Text

storage_context = storage_context.from_defaults()

index = GPTVectorStoreIndex.feom_documents(docs, storage_context=storage_context)

# save
index.storage_context.persist(persist_dir="./dir")

# load
from llama_index import load_index_from_storage

storage_context = StorageContext.from_defaults(persist_dir="./dir")

loaded_index = load_index_from_storage(storage_context)

᲼᲼᲼᲼᲼᲼᲼᲼᲼᲼᲼᲼

True though i wanted to do it this way :

Plain Text

embed_model = LangchainEmbedding(HuggingFaceEmbeddings())
llm_predictor = LLMPredictor(llm=OpenAI(temperature=0, model_name="text-davinci-002"))
persist_dir = "stores"
storage_context = StorageContext.from_defaults(
    docstore=SimpleDocumentStore.from_dict({}),
    vector_store=SimpleVectorStore.from_dict({}),
    index_store=SimpleIndexStore.from_dict({}),
)
service_context = ServiceContext.from_defaults(embed_model=embed_model, llm_predictor=llm_predictor)
documents = BeautifulSoupWebReader().load_data(urls)

So it looks real cleaan y'know, i think that's how people would do it too

Add a reply

Find answers from the community

Persist dir