Find answers from the community

Updated 3 months ago

I'm a bit confused about the metadata

I'm a bit confused about the metadata generation of MilvusVectorStore.query.

My actual problem is that I want to specify a text field and some fields that are added to the metadata without adding the text itself to the metadata. If I only set the output_fields in the MilvusVectorStore constructor, I need to include the text field (or it won't be queried at all), but that also adds it to the metadata.

The output_fields parameter passed to the milvus client can either be [*], query.output_fields or self.output_fields base.py L415. When the metadata is generated, it always uses self.output_fields base.py L464.

Does that mean that I could set the output_fields in the constructor to only the metadata fields and then somehow set query.output_fields to the metadata fields plus the text field? I'm new to llamaindex so I don't know if this is the correct behavior but wouldn't it make sense to always exclude the text field from the metadata if it is configured? And shouldn't the metadata always use the output_fields that are actually passed to the milvus client?
R
o
8 comments
you can explicitely ask it to exclude some metadata
thanks, i'll try that tomorrow
I'm not creating the document myself so I'll have to figure out where i can set that
try it out and let me know
I tried it now and it works, I added a node_postprocessor to my chat engine.
Plain Text
class MetadataFilter(BaseNodePostprocessor):
    def _postprocess_nodes(self, nodes, query_bundle=None):
        for node in nodes:
            node.node.excluded_llm_metadata_keys = ['text']
        return nodes

Now I'd just have to read the custom text_key from my settings instead. I'll wait to see what comes out of the PR I opened about this, but otherwise this seems like a good solution, thanks.
Add a reply
Sign up and join the conversation on Discord