Has anyone encoutered a similar problem

QdrantReader pydantic error:
I am ingesting documents into qdrant using this code:

Plain Text

 qdrant_client = GetQdrantClient().get_client()
            logger.debug("creating reader")
            reader = SimpleDirectoryReader(
                input_files=[f"./temp/{user_id}/{file_name}"]
            )
            documents = reader.load_data()
            qdrant_vector_store = QdrantVectorStore(
                client=qdrant_client, collection_name=COLLECTION_NAME
            )
            pipeline = IngestionPipeline(
                transformations=[AZURE_EMBEDDING], vector_store=qdrant_vector_store
            )
            for document in documents:
                document.metadata["user_id"] = user_id
                document.metadata["file_name"] = file_name
            nodes = pipeline.run(documents=documents, num_workers=4)
            return

12 comments

ddrButts

and when I try and retrieve a list of documents using QdrantReader like this:

Plain Text

          logger.debug("Loading data from Qdrant")
            documents = reader.load_data(
                collection_name=collection_name,
                query_vector=query_vector,
                should_search_mapping=metadata_filter,
                must_search_mapping=metadata_filter,
            )
            reader = reader.load_data(
                collection_name="doc_chat",
                query_vector=[0.0, 0.0, 0.0],
                should_search_mapping={"user_id": user_id},
                must_search_mapping={"user_id": user_id},
            )
            logger.info(f"Retrieved documents: {documents}")
            index = SummaryIndex.from_documents(documents)
            # set Logging to DEBUG for more detailed outputs
            query_engine = index.as_query_engine()
            response = query_engine.query("")
            logger.info(f"Query response: {response}")
            return documents       
And I get this error:1 validation error for Document: extra_info none is not an allowed value (type=type_error.none.not_allowed)

LLogan M

ooo qdrant reader is definitely not maintend well lol

LLogan M

I would guess that your first could example would work though

LLogan M

curious what the full traceback is there

ddrButts

I am not set on QdrantReader but just want to pull a list of all the docsuments based oo metadat filtering by the user_id key.. Is there another reader or method that you know that can do this?

ddrButts

the first code snippet works and puts the data in qdrant, but it is obviously misisng some metadata fields

ddrButts

or missing something thet is in the Document type

LLogan M

Yea the second thing seems to be a small bug in the qdrant reader

You could just do

Plain Text

retriever = index.as_retriever(filters=filters, similarity_top_k=10000)
nodes = retriever.retrieve("query string")

ddrButts

I thought of that but I don't have a query string here, I just want to see if I can pul a list of docs in qdrant that have a specific metadata key

ddrButts

I am aiming for multi-tenancy rag, where a user can choose which docs to rag on, but have a list of the docs inserted perviously rendered to a ui

ddrButts

so I was afraid a similarity search wouldm't do the trick here.

LLogan M

if you set the top-k to a very large number, it should return everything

Alternatively, you can use qdrants api directly too

Add a reply

Find answers from the community

Has anyone encoutered a similar problem