Its implemented for chroma, qdrant, and simple vector store (the default)
You might have to upgrade chroma: pip install -U llama-index-vector-stores-chroma
(If that doesn't work, might have missed the publish)
@Logan M ill upgrade both and come back to you
@Logan M okay so clear() is now working. However, the delete function apart of chroma isn't working now specifically this function:
def clear(self) -> None:
"""Clear the collection."""
ids = self._collection.get()["ids"]
self._collection.delete(ids=ids)
its giving this response:
""" You must provide either ids, where, or where_document to delete. If you want to delete all data in a collection you can delete collection itself using the delete_collection method. Or alternatively,
you can get() all the relevant ids and then delete them. """
script for context:
db3 = chromadb.PersistentClient(path="/home/headquarters/Documents/Guardian/Storages/ChatMemory/")
chroma_collection = db2.get_or_create_collection(collection_name)
vector_store = ChromaVectorStore(chroma_collection=chroma_collection)
vector_memory = VectorMemory.from_defaults(
vector_store=vector_store, # leave as None to use default in-memory vector store
embed_model=Settings.embed_model,
retriever_kwargs={"similarity_top_k": 1},
)
# let's set some initial messages in our secondary vector memory
system = ChatMessage.from_str(content="You are a SOMEWHAT helpful assistant.", role="system")
Composable_sys = ChatMessage.from_str(content="You are a REALLY helpful assistant.", role="system")
question = ChatMessage.from_str(role="user", content=query)
response = ChatMessage.from_str(role="assistant", content=str(response))
vector_memory.set([system, question, response])
chat_memory_buffer = ChatMemoryBuffer.from_defaults()
composable_memory = SimpleComposableMemory.from_defaults(
primary_memory=chat_memory_buffer,
secondary_memory_sources=[vector_memory],
)
# load into all memory sources modules"
composable_memory.put(Composable_sys)
composable_memory.put(question)
composable_memory.put(response)
msgs = composable_memory.get(query)
print(msgs[0])lol but it did provide ids
Unless ids was empty?
There's even a unit test for this..
would roles in the chat message be the ids it discusses
obviously its new code so im not worried about issues erupting but wondering if its on my end or if I need to edit a package
def test_clear(
vector_store: ChromaVectorStore, node_embeddings: List[TextNode]
) -> None:
vector_store.add(node_embeddings)
vector_store.clear()
res = vector_store.get_nodes(
node_ids=[
"c330d77f-90bd-4c51-9ed2-57d8d693b3b0",
"c3d1e1dd-8fb4-4b8f-b7ea-7fa96038d39d",
"c3ew11cd-8fb4-4b8f-b7ea-7fa96038d39d",
]
)
assert len(res) == 0Yea not sure, I'd have to dig in a bit 👀
@Logan M I'll try making adjustments to that script but I'll dig in as well. I was messing a lot with the VectorMemory() package prior to updating Chromadb. God Bless open source.
maybe theres another approach with a second package acting as one of VectorM() dependants
i see its a test to see if it clears without responding back to with an error
@Logan M still having problems with it. The issue is I don't know the node ids in the history because when it resets it doesn't have anything stored.
I also tried calling the metadata so it would pull all nodes aligned with sub_dicts like how its stored and Im having issues trying to use metadata filters with it becuase it requires "ids"
Hmm, I feel like I'm lost now on what the issue is.
Are you able to replicate on a Google colab? Probably more useful to have something to iterate on
@Logan M heres the google colab
Ok cool, I will take a look at this
the goal of the code is to make short term history and long like the composable is built for but automating the message upload process so I don't have to type out all the facts first
@Logan M did you ever figure out what happened with the code or figure out changes that needed to be made?
or did it run perfect for you XD
Oof I haven't had time to look yet lol was on vacation this weekend
Will fund some time today probably 🙏
@Logan M Oh shit all good. Hope vacation was good. Sorry felt bad for pressing just didn't know if you were still looking
appreciate all ur help regardless so take your time
rather get it right then somewhat right
@Logan M let me know if youve figured anything out eagerly awaiting potential changes😁
I straight up dont think it will work with chromadb lol
chromadb doesn't let you put dicts in the metadata
maybe the vector memory should be using json.dumps to allow it to be more generic
@Logan M Ah all good what would you recommend using instead of Chroma then?
I'm trying to find a solution to a persistent local database that can hold composable history
@Logan M is there a solution for persistent composable memory or at least persistent vector memory outside of Qdrant and ChromaDB worth checking?
Just a name of a similar software would help just need a new angle on the problem
I think I just need to update core so that the memory serializes better.
For other vector stores to support this, they need to implement those new methods added to the base class (PRs welcome there for sure)
