Find answers from the community

Updated 3 months ago

while retrieving text using vector

while retrieving text using vector similarity, i am seeing irrelavant text which has some similarity also getting pulled in the context, i m using nodes with entity, is there a way to filter out such nodes/text

7 comments

WWhiteFang_Jr

You can try using similarity post processor to set a threshold for similarity cut off below which all the nodes will not get picked.

https://docs.llamaindex.ai/en/stable/module_guides/querying/node_postprocessors/node_postprocessors.html#similaritypostprocessor

aanupamaze

Thank you for the advice Actually I have tried doing that but I see that similarity scores are some times higher for irrelevant text

aanupamaze

Bringing in entities helped bridge that gap

aanupamaze

But still it’s far from optimal

LLogan M

That's just the nature of how embeddings work sometimes. Maybe try using a re-ranker

aanupamaze

Yes I have used reranker too

aanupamaze

Is there a way that if I create extract entities for nodes in a different way then the comparison becomes better

Add a reply