Find answers from the community

Updated 10 months ago

GitHub - wenqiglantz/edd-recursive-doc-a...

I'm using almost the exact same code as with the EDD notebooks from this repo
Basically I'm doing this:
Plain Text
from llama_index.core.evaluation import DatasetGenerator, FaithfulnessEvaluator, RelevancyEvaluator, BatchEvalRunner

evaluator_llm = OpenAI(model="gpt-4", temperature=0)
faithfulness_evaluator = FaithfulnessEvaluator(llm=evaluator_llm)
relevancy_evaluator = RelevancyEvaluator(llm=evaluator_llm)

eval_runner = BatchEvalRunner(
    {"faithfulness": faithfulness_evaluator, "relevancy": relevancy_evaluator},
    workers=6,
    show_progress=True,
)

eval_results = await eval_runner.aevaluate_queries(
    query_engine=hybrid_query_engine, queries=subsapmple_synth_questions
)
a
s
L
25 comments
hmmm... is there more traceback you can share? I'm trying to find out what is trying to invoke search_batch
This is the whole error:
looks like its QdrantVectorStore._aclient which is None
Ok, reading the error now, apparently is because I haven't initialized Qdrant to support async queries
Now when initializing an async Qdrant VectorStoreIndex I'm getting this error
It says that the object does not have a get_collection attribute
Looks like there is still issues with defining the client, as _client (or _aclient or both) are still None
Is this an issue with 0.10 version? Becouse it is in the docs, apparently it was working previously
I think you need to provide both aclient and client, if you aren't already
(note that if you are using :memory:, the two clients do not share data...)
How do I provide the aclient?
Plain Text
import qdrant_client

client = qdrant_client.QdrantClient(...)
aclient = qdrant_client.AsyncQdrantClient(...)
So, I'll have to create 2 indexes if using memory or disc? One for sync and one for async?
qdrant doesn't let you connect to them at the same time hey? :PSadge:
They should really improve that...
I'm actually not sure, will try it right now
But thanks for the answers! You both were really helpful @nerdai @Logan M
Plain Text
client = qdrant_client.QdrantClient(
    # you can use :memory: mode for fast and light-weight experiments,
    # it does not require to have Qdrant deployed anywhere
    # but requires qdrant-client >= 1.1.1
    location=":memory:"
    # otherwise set Qdrant instance address with:
    # uri="http://<host>:<port>"
    # set API KEY for Qdrant Cloud
    # api_key="<qdrant-api-key>",
)

from https://docs.llamaindex.ai/en/stable/examples/vector_stores/QdrantIndexDemo.html
I was following that, but apparently is how @Logan M stated, I just saw that QdrantVectorStore can receive both a syncronous client and an async one. I'm testing passing both of them now
Attachment
image.png
As a follow up, for local testing, only using :memory: for both client and aclient will work, since you will not be able to use your disk storage if you want to access concurrently the index
But again, thank you both @Logan M @nerdai you were really helpful
I was able to run my evaluations when changing all to async
Add a reply
Sign up and join the conversation on Discord