Storage

MMurk418

Hey! Would you happen to know if its possible to load a VectorStoreIndex by passing the docstore, graph_store, index_store, and vector_store directly? I'm currently downloading the files from a GCS bucket, which means that by default I would need to download them to a local directory, before loading the index from the persist_dir. But I was wondering if it would be possible to instead just pass the docstore, graph_store, index_store, and vector_store directly after downloading them into memory from the storage bucket? Any insights would be much appreciated. Thanks again!

2 comments

LLogan M

Yea, you could pass it in directly to the storage context

`StorageContext.from_defaults(docstore=docstore, index_store=index_store, ...)

index = VectorStoreIndex([], storage_context=storage_context)
```

Although if you are using Google cloud buckets, I think there is support for fsspec integration
https://gpt-index.readthedocs.io/en/stable/core_modules/data_modules/storage/save_load.html#using-a-remote-backend

MMurk418

Thanks, I tried the "StorageContext.from_dict" method as well. And I've used gcsfs as the remote backend in the past. But I'm realizing that my bottleneck is probably network latency rather than the time it takes to load the index lol, cause from what I can tell the index actually loads pretty fast. Thanks!

Add a reply

Find answers from the community

Storage