Sparse Retriever: Considering Text or Metadata in Retrieval

Question

sparse_retriever = BM25Retriever.from_defaults(docstore=docstore, similarity_top_k=5)Does BM25Retriever takes metadata of nodes into account or not? If yes then how to prevent it and make it to only consider text of nodes into account.

WhiteFang_Jr · Answer

You can do that when you documents are created. Once your documents are created , each document contains fields that gets ignored while creating the embeddings or being sent to LLM. doc = Document(text = "This is first docu") # if you add any metadata key here it will be excluded from embedding process
doc.excluded_embed_metadata_keys = []
# same here , any key added here will not go to LLM for response generation
doc.excluded_llm_metadata_keys = []

Find answers from the community

Sparse Retriever: Considering Text or Metadata in Retrieval