Find answers from the community

Updated 3 months ago

Hey how can I exclude metadata with None

Hey, how can I exclude metadata with None values when sending the node content to GPT? For example, I have a "name" attribute as metadata, and nodes without a specified name have an empty value for this attribute (None by default).
So, sometimes, when I ask about the name and the retriever extracts nodes with name = None, GPT returns None instead of "I don't know," for example.
By the way, I'm using version "0.6.31."
L
Y
6 comments
I think you could write either a custom retriever or custom node-postprocessor to fix those nodes πŸ€”
I was thinking about that too, maybe a custom node-postprocessor, so by doing that I can update the text that will be sent to gpt (just delete those metadata from text) or I have to delete those nodes completely ?
Is there an example of creating a custom node postprocessor available on the doc or github?
I think you could just remove those keys from the metadata if they are None πŸ€”

Not a full example, but the API is super simple: https://gpt-index.readthedocs.io/en/latest/how_to/query_engine/advanced/second_stage.html#api-interface

Example (note that in 0.6.34+, extra_info changed to metadata, but I'll use the old name here)
Plain Text
class MetadataPostprocessor:
    """Metadata Node postprocessor."""

    def postprocess_nodes(
        self, nodes: List[NodeWithScore], query_bundle: Optional[QueryBundle]
    ) -> List[NodeWithScore]:
        """Postprocess nodes."""
        for n in nodes:
            metadata = n.node.extra_info
            if metadata and 'name' in metadata and metadata['name'] is None:
                del n.node.extra_info['name']
        return nodes
Thank you!
Then, I just need to append it to node_postprocessors like the following code ?
query_engine = RetrieverQueryEngine.from_args( retriever, node_postprocessors=[ SimilarityPostprocessor(similarity_cutoff=similarity_cutoff), MetadataPostprocessor(), ], )
Yea exactly! The node_postprocessors should be applied in the order they are given πŸ™‚
Add a reply
Sign up and join the conversation on Discord