Find answers from the community

Updated 8 months ago

llama_index/llama-index-core/llama_index...

does nayone know how to custom models for PII masking
https://github.com/run-llama/llama_index/blob/b430d1757081e1944474bf281cd757e69431cc07/llama-index-core/llama_index/core/postprocessor/pii.py#L96

i can find where to pass the bert model and i dont know how the default defaulted to dbmdz/bert-large-cased-finetuned-conll03-english and revision f2482bf (https://huggingface.co/dbmdz/bert-large-cased-finetuned-conll03-english). is being supplse

5 comments

I think its hardcoded

Plain Text

from transformers import pipeline  # pants: no-infer-dep

ner = pipeline("ner", grouped_entities=True)

This just loads the model huggingface thinks is best for ner

i created a custom node processor

yea that works

I thinks its better than the default

Attachment

Add a reply