I'm working with Qdrant and LLamaIndex.

eerizvi

I'm working with Qdrant and LLamaIndex. I have a bunch of release note documents in html format that I was able to parse using UnstructuredElementNodeParser and persist to Qdrant Vector DB via vectorestoreindex.

However, the user can ask for all the changes between two dates in which case I need to retrieve relevant documents in between the date range so the process is two folds: 1) try to determine which relevant documents to query. 2) query those documents for more context.

I'm using chat_engine with modes "openai" and "context" none seem to do this. Also it seems that the vector store doesn't really store document names as that would be helpful since the document names contain their release dates.

Is there a way to do this type of hierarchical search? would I be able to use recursiveretriever for this purpose and if so how would I configure it and work with a chat-engine?

3 comments

rrahul

Yeah you should be able to use Recursive Reader for this. Here is an outline of how you can use Recursive Reader for this:

Plain Text

from llama_index.indices.vector_store.retrievers import RecursiveRetriever, BaseRetriever
from datetime import datetime

# Custom retriever to filter documents by date range
class DateRangeRetriever(BaseRetriever):
    def retrieve(self, query, params=None):
        # Extract date range from the query
        start_date, end_date = extract_date_range(query)
        # Query Qdrant with the date range metadata to get document IDs
        document_ids = query_qdrant_by_date_range(start_date, end_date)
        return document_ids

# Function to extract date range from the query
def extract_date_range(query):
    # Implement logic to extract date range from the query
    # For example, use regex or natural language processing
    return start_date, end_date

# Function to query Qdrant by date range
def query_qdrant_by_date_range(start_date, end_date):
    # Implement logic to query Qdrant with the date range
    # Return a list of document IDs
    return document_ids

# Set up the RecursiveRetriever
date_range_retriever = DateRangeRetriever()
context_retriever = ... # Set up your retriever for querying documents for context
recursive_retriever = RecursiveRetriever(date_range_retriever, context_retriever)

# Integrate with the chat engine
# You will need to modify or extend the chat engine to use the recursive_retriever

LLogan M

You could also attach the dates as metadata, and use an auto retriever, so that the LLM writes metadata filters on the fly

eerizvi

Thanks @rahul @Logan M I will try this out

Add a reply

Find answers from the community

I'm working with Qdrant and LLamaIndex.