Find answers from the community

Updated 6 months ago

Bm25

At a glance

The community member is experiencing a ZeroDivisionError when trying to use the BM25Retriever with a Weaviate vector store. The comments suggest that the BM25Retriever requires a docstore or nodes, and when using Weaviate (or other vector databases), the docstore is disabled by default. The community members recommend passing the nodes directly to the BM25Retriever instead. They also discuss the possibility of retrieving all the nodes from the index, and the use of Qdrant for hybrid or fusion search, where the community member can implement their own custom ranking function.

Useful resources
Hi, while trying to use bm25_retriever = BM25Retriever.from_defaults(
docstore=index.docstore, similarity_top_k=2
), getting this error - ZeroDivisionError
I am using weaviate as vector store and the loaded index does have a good amount of documents, How to resolve this error?
L
P
5 comments
Bm25 requires a docstore or nodes.

When using weaviate (or other vector db integrations) the docstore is disabled by default to simplify storage

Try passing in nodes directly to the bm25 receiver instead?
is it possible to pass all the nodes of an index?
Only if you still have the nodes from when you created the index. Or maybe do a retrieve with the top k set to a large number lol
Hey, which vector DB to use for doing hybrid or fusion search? I need something around which I can build my own customised ranking
I did just release sparse embedding support for qdrant (which is a type of hybrid search)
https://docs.llamaindex.ai/en/stable/examples/vector_stores/qdrant_hybrid.html

You can implement your own reranking function for when you fuse sparse and dense results
Add a reply
Sign up and join the conversation on Discord