Hey how can I exclude metadata with None

YYasmine

Hey, how can I exclude metadata with None values when sending the node content to GPT? For example, I have a "name" attribute as metadata, and nodes without a specified name have an empty value for this attribute (None by default).
So, sometimes, when I ask about the name and the retriever extracts nodes with name = None, GPT returns None instead of "I don't know," for example.
By the way, I'm using version "0.6.31."

6 comments

LLogan M

I think you could write either a custom retriever or custom node-postprocessor to fix those nodes 🤔

YYasmine

I was thinking about that too, maybe a custom node-postprocessor, so by doing that I can update the text that will be sent to gpt (just delete those metadata from text) or I have to delete those nodes completely ?

YYasmine

Is there an example of creating a custom node postprocessor available on the doc or github?

LLogan M

I think you could just remove those keys from the metadata if they are None 🤔

Not a full example, but the API is super simple: https://gpt-index.readthedocs.io/en/latest/how_to/query_engine/advanced/second_stage.html#api-interface

Example (note that in 0.6.34+, extra_info changed to metadata, but I'll use the old name here)

Plain Text

class MetadataPostprocessor:
    """Metadata Node postprocessor."""

    def postprocess_nodes(
        self, nodes: List[NodeWithScore], query_bundle: Optional[QueryBundle]
    ) -> List[NodeWithScore]:
        """Postprocess nodes."""
        for n in nodes:
            metadata = n.node.extra_info
            if metadata and 'name' in metadata and metadata['name'] is None:
                del n.node.extra_info['name']
        return nodes

YYasmine

Thank you!
Then, I just need to append it to node_postprocessors like the following code ?

query_engine = RetrieverQueryEngine.from_args(
    retriever,
    node_postprocessors=[
        SimilarityPostprocessor(similarity_cutoff=similarity_cutoff),
        MetadataPostprocessor(),
    ],
)

LLogan M

Yea exactly! The node_postprocessors should be applied in the order they are given 🙂

Add a reply

Find answers from the community

Hey how can I exclude metadata with None