Does Persistent Client saves data to VRAM of GPU?

I have a question, when loading an index from a persistent chromadb vectorstore , is it loaded in the vram of gpu ?

5 comments

No, when you use the persist to save index, it's saved onto your disk somewhere so you no need to load it again, VRAM is volatile memmory, so once you turn off your computer, the data would be gone.

gghxsted.

no i meant after persisting it to disk, and loading it again, does it get loaded in vram or ram ?

gghxsted.

Plain Text

chroma_client = chromadb.PersistentClient(path=chroma_persistent_dir)
chroma_collection = chroma_client.get_collection(collection_name)
vector_store = ChromaVectorStore(chroma_collection=chroma_collection)
llm = llm = Ollama(model="llama3:70b-instruct", request_timeout=3000.0)
embed_model = HuggingFaceEmbedding(model_name="Alibaba-NLP/gte-large-en-v1.5", trust_remote_code=True, embed_batch_size=2)
Settings.llm = llm
Settings.embed_model = embed_model
index = VectorStoreIndex.from_vector_store(vector_store, embed_model=embed_model)

the chromadb is already saved to disk, all i'm doing is loading the collection and creating an index with VectorStoreIndex, and wanted to know if it actually loads the index in the vram of the gpu, since i don't have alot of vram left from loading a big model + embedding model

gghxsted.

@Logan M do you have any insight about this pls ?

LLogan M

I have no idea, it depends how chroma is implemented. I'm not even sure if it uses vram, just normal ram I think

Add a reply

Find answers from the community

Does Persistent Client saves data to VRAM of GPU?