I'm a bit confused about the metadata

oowlbot

I'm a bit confused about the metadata generation of MilvusVectorStore.query.

My actual problem is that I want to specify a text field and some fields that are added to the metadata without adding the text itself to the metadata. If I only set the output_fields in the MilvusVectorStore constructor, I need to include the text field (or it won't be queried at all), but that also adds it to the metadata.

The output_fields parameter passed to the milvus client can either be [*], query.output_fields or self.output_fields base .py L415. When the metadata is generated, it always uses self.output_fields base .py L464.

Does that mean that I could set the output_fields in the constructor to only the metadata fields and then somehow set query.output_fields to the metadata fields plus the text field? I'm new to llamaindex so I don't know if this is the correct behavior but wouldn't it make sense to always exclude the text field from the metadata if it is configured? And shouldn't the metadata always use the output_fields that are actually passed to the milvus client?

8 comments

RRoland Tannous

you can explicitely ask it to exclude some metadata

RRoland Tannous

check here

RRoland Tannous

https://docs.llamaindex.ai/en/latest/module_guides/loading/documents_and_nodes/usage_documents/#customizing-llm-metadata-text

oowlbot

thanks, i'll try that tomorrow
I'm not creating the document myself so I'll have to figure out where i can set that

RRoland Tannous

try it out and let me know

oowlbot

I tried it now and it works, I added a node_postprocessor to my chat engine.

Plain Text

class MetadataFilter(BaseNodePostprocessor):
    def _postprocess_nodes(self, nodes, query_bundle=None):
        for node in nodes:
            node.node.excluded_llm_metadata_keys = ['text']
        return nodes

Now I'd just have to read the custom text_key from my settings instead. I'll wait to see what comes out of the PR I opened about this, but otherwise this seems like a good solution, thanks.

RRoland Tannous

you're welcome

Add a reply

Find answers from the community

I'm a bit confused about the metadata