Topic: nested `MetadataFilters`

hhaedamon

Topic: nested MetadataFilters
I am attempting to use MetadataFilters (similar to how this article describes: https://docs.llamaindex.ai/en/stable/examples/multi_tenancy/multi_tenancy_rag.html)
Recently, nested MetadataFilters were introduced in version 0.10.19
https://github.com/run-llama/llama_index/pull/11778
which fixes:

https://github.com/run-llama/llama_index/issues/10353

***
Notably. this changes the code in llama_index.core.vector_stores.types.py::MetadataFilters:

Plain Text

class MetadataFilters(BaseModel):
    """Metadata filters for vector stores."""

    # Exact match filters and Advanced filters with operators like >, <, >=, <=, !=, etc.
    filters: List[Union[MetadataFilter, ExactMatchFilter, "MetadataFilters"]]

However, when I run the code below, I encounter the following errors

9 comments

LLogan M

I think nested metadata filters are only handled by the postgres vector store 🤔

hhaedamon

But the error is happening prior to any attempts to persist the data to any vector store. Simply trying to run a query results in the exception

LLogan M

What is the error? I don't remeber if you posted it)

hhaedamon

It's posted just below my original post (with more context)
Running r = engine.query("some question") yields

Plain Text

File /path/to/lib/python3.10/site-packages/llama_index/core/vector_stores/types.py:170, in MetadataFilters.legacy_filters(self)
    166 for filter in self.filters:
--> 170     if filter.operator != FilterOperator.EQ:
    171         raise ValueError(
    172             "Vector Store only supports exact match filters. "
    173             "Please use ExactMatchFilter or FilterOperator.EQ instead."
    174         )
    175     filters.append(ExactMatchFilter(key=filter.key, value=filter.value))

AttributeError: 'MetadataFilters' object has no attribute 'operator'

hhaedamon

However I think you might be correct, VectorStoreIndex's default storage_context is SimpleDocumentStore and the PR is just for PGVectorStore

I thought it would be a more intuitive design if the engine's representation was decoupled from it's persistence-backend/storage-context

Since the MetadataFilters class is now defined as:

Plain Text

class MetadataFilters(BaseModel):
    filters: List[Union[MetadataFilter, MetadataFilters]]

one would assume that it just "works" for other storage-contexts... other ppl will run into this issue

hhaedamon

@Logan M I'm just slowly beginning to go through the underlying llama-index codebase - can you confirm if nested metadata filters have to be implemented independently for every storage-context?!

LLogan M

they have to be implemented for each vector store yes. Since they all have their own filtering logic

LLogan M

seems like an issue here with using nested filters + the older legacy operation (not every vector db has moved off of legacy)

LLogan M

its a lot of work :PSadge:

Add a reply

Find answers from the community

Topic: nested `MetadataFilters`