Topic: nested `MetadataFilters`

At a glance

The community member is attempting to use nested MetadataFilters in the llama_index library, which was recently introduced in version 0.10.19. However, when running the code, they encounter an error related to the MetadataFilters class not having an operator attribute.

The comments suggest that nested metadata filters are only handled by the Postgres vector store, and the error is occurring before any attempts to persist data to a vector store. One community member notes that the MetadataFilters class is now defined to accept a list of MetadataFilter, ExactMatchFilter, or MetadataFilters objects, which suggests it should "just work" for other storage contexts, but this is not the case.

Another community member confirms that nested metadata filters have to be implemented independently for each vector store, as they all have their own filtering logic. This seems to be an issue with using nested filters and the older legacy operation, as not every vector database has moved off of the legacy implementation.

There is no explicitly marked answer in the comments, but the community members discuss the challenges of implementing nested metadata filters across different vector store implementations.

Useful resources

hhaedamon

Topic: nested MetadataFilters
I am attempting to use MetadataFilters (similar to how this article describes: https://docs.llamaindex.ai/en/stable/examples/multi_tenancy/multi_tenancy_rag.html)
Recently, nested MetadataFilters were introduced in version 0.10.19
https://github.com/run-llama/llama_index/pull/11778
which fixes:

https://github.com/run-llama/llama_index/issues/10353

***
Notably. this changes the code in llama_index.core.vector_stores.types.py::MetadataFilters:

Plain Text

class MetadataFilters(BaseModel):
    """Metadata filters for vector stores."""

    # Exact match filters and Advanced filters with operators like >, <, >=, <=, !=, etc.
    filters: List[Union[MetadataFilter, ExactMatchFilter, "MetadataFilters"]]

However, when I run the code below, I encounter the following errors

9 comments

LLogan M

I think nested metadata filters are only handled by the postgres vector store 🤔

hhaedamon

But the error is happening prior to any attempts to persist the data to any vector store. Simply trying to run a query results in the exception

LLogan M

What is the error? I don't remeber if you posted it)

hhaedamon

It's posted just below my original post (with more context)
Running r = engine.query("some question") yields

Plain Text

File /path/to/lib/python3.10/site-packages/llama_index/core/vector_stores/types.py:170, in MetadataFilters.legacy_filters(self)
    166 for filter in self.filters:
--> 170     if filter.operator != FilterOperator.EQ:
    171         raise ValueError(
    172             "Vector Store only supports exact match filters. "
    173             "Please use ExactMatchFilter or FilterOperator.EQ instead."
    174         )
    175     filters.append(ExactMatchFilter(key=filter.key, value=filter.value))

AttributeError: 'MetadataFilters' object has no attribute 'operator'

hhaedamon

However I think you might be correct, VectorStoreIndex's default storage_context is SimpleDocumentStore and the PR is just for PGVectorStore

I thought it would be a more intuitive design if the engine's representation was decoupled from it's persistence-backend/storage-context

Since the MetadataFilters class is now defined as:

Plain Text

class MetadataFilters(BaseModel):
    filters: List[Union[MetadataFilter, MetadataFilters]]

one would assume that it just "works" for other storage-contexts... other ppl will run into this issue

hhaedamon

@Logan M I'm just slowly beginning to go through the underlying llama-index codebase - can you confirm if nested metadata filters have to be implemented independently for every storage-context?!

LLogan M

they have to be implemented for each vector store yes. Since they all have their own filtering logic

LLogan M

seems like an issue here with using nested filters + the older legacy operation (not every vector db has moved off of legacy)

LLogan M

its a lot of work :PSadge:

Add a reply

Find answers from the community

Topic: nested `MetadataFilters`