explain the semantic search and hybrid search

@kapa.ai explain the semantic search and hybrid search

5 comments

Please stand by @maximilian as I search the knowledge base. We are experiencing especially high traffic right now and might take longer than usual. :wave:

kkapa.ai

----------------------
Semantic search is a basic usage of LlamaIndex that allows you to search for relevant information in your data using natural language queries. It uses a vector store to perform the search. Here's an example of how to perform a semantic search using LlamaIndex:

Plain Text

from llama_index import GPTVectorStoreIndex, SimpleDirectoryReader
documents = SimpleDirectoryReader('data').load_data()
index = GPTVectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine()
response = query_engine.query("What did the author do growing up?")
print(response)

Source: Queries over your Data (https://gpt-index.readthedocs.io/en/latest/use_cases/queries.html)

Hybrid search combines keyword lookup retrieval with vector retrieval using "AND" and "OR" conditions. It allows you to perform more complex searches by combining the results of both keyword and vector-based searches. Here's an example of how to define a custom retriever class that can implement basic hybrid search with both keyword lookup and semantic search:

Plain Text

from llama_index import QueryBundle
from llama_index.data_structs import NodeWithScore
from llama_index.retrievers import BaseRetriever, VectorIndexRetriever, KeywordTableSimpleRetriever
from typing import List

Source: Retriever Query Engine with Custom Retrievers - Simple Hybrid Search (https://gpt-index.readthedocs.io/en/latest/examples/query_engine/CustomRetrievers.html)
----------------------
Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:
----------------------

mmaximilian

@kapa.ai how do i create a KeywordTableSimpleRetriever

kkapa.ai

Let me check the knowledge base for you. One moment! :books:

kkapa.ai

----------------------
To create a KeywordTableSimpleRetriever, you need to first initialize a BaseGPTKeywordTableIndex. Then, you can create the KeywordTableSimpleRetriever using the index. Here's an example:

Plain Text

from llama_index.indices.keyword_table import GPTSimpleKeywordTableIndex
from llama_index.retrievers import KeywordTableSimpleRetriever

# Initialize the keyword table index
keyword_index = GPTSimpleKeywordTableIndex.from_documents(documents)

# Create the KeywordTableSimpleRetriever
keyword_retriever = KeywordTableSimpleRetriever(index=keyword_index)

In this example, documents is a list of documents that you want to index. You can load the documents using a SimpleDirectoryReader as shown in the previous answer.

Source: Keyword Table Retrievers (https://gpt-index.readthedocs.io/en/latest/reference/query/retrievers/table.html)
----------------------
Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:
----------------------

Add a reply

Find answers from the community

explain the semantic search and hybrid search