Find answers from the community

Updated 10 months ago

Questions

Hey if I have a an array of a thousand questions and I want to search for similarity to a specific question - what's the best way to approach this? Add the bank to a chromadb with metadata and then do a search?

13 comments

LLogan M

Yea pretty much 👍

vvalu

so i make a document out of each question - put the question number/id as the metadata

vvalu

use a local embedding model to make an index

vvalu

then query

vvalu

will this do both vector and bip search?

LLogan M

What is bip search? But otherwise yes

vvalu

like keyword search

vvalu

it seems the default vectorstoreindex doesnt have it

vvalu

i should use chromadb?

LLogan M

Yea combing vector and keyword I usually referred to as hybrid search (at least lately in this space)

Only a few dbs have it built in tbh. Other times you have to setup yourself (coming bm25 with a vector index for example)

LLogan M

https://docs.llamaindex.ai/en/stable/module_guides/storing/vector_stores.html#vector-store-options-feature-support

LLogan M

From scratch with sparse vectors + qdrant
https://docs.llamaindex.ai/en/stable/examples/vector_stores/qdrant_hybrid.html#qdrant-hybrid-search

Using builtin with weaviate
https://docs.llamaindex.ai/en/stable/examples/vector_stores/WeaviateIndexDemo-Hybrid.html

Building your own with bm25
https://docs.llamaindex.ai/en/stable/examples/retrievers/reciprocal_rerank_fusion.html#reciprocal-rerank-fusion-retriever

vvalu

used the tutorial to build thx

Add a reply