Mitch 🪬

Resolving Memory Leaks in Qdrant Vector Database Connections

Hi guys, hope everything is going well. I'm having this issue where after asking my model a couple of questions, it returns this error and won't run again until I restart the system:

Plain Text

Both client and aclient are provided. If using `:memory:` mode, the data between clients is not synced.

My guess is that there may be a memory leak somewhere within the system, likely within the connection to my Qdrant vector database. The main solution I've been seeing is to try implementing AsyncQdrantClient but I'm not sure where to add it in - is this an issue that's occurring directly within my RAG workflow or does it go deeper and should be applied in a different section? Any advice would be appreciated!

5 comments

MMitch 🪬

Modifying the Reranking System in LLMRerank

Hi guys, quick question about modifying the reranking system. I implemented a couple changes into _postprocess_nodes inside of LLMRerank where it gives priority to nodes with a more recent date and a specific subtype. However, since llm_rerank.py is directly part of a Python package, I'm having trouble with utilizing the modified file instead of the original version of LLM rerank. Is there either a way to make these same node reranking adjustments in my workflow file or override the original package with my new code? Here is my updated section of the code:

Plain Text

for node, relevance in zip(choice_nodes, relevances):
                # date score
                date_str = node.metadata.get('date')
                date_score = datetime.strptime(date_str, "%Y-%m-%d").timestamp()
                # Subtype score
                subtype = node.metadata.get('subtype', '')
                subtype_score = 1.0 if subtype == "pro report" else 0.0
                node_score = relevance + date_score + subtype_score
                initial_results.append(NodeWithScore(node=node, score=node_score))

1 comment

MMitch 🪬

Integrating Grok Into Llm Reranking System

Hi all, I am working on my RAG workflow for my model and need some guidance. I want to integrate Grok into my LLM reranking system but it's not working for me. Grok's documentation says it's compatible with the OpenAI SDK so I am assuming this is the correct format, but I still can't get it to run:

ranker = LLMRerank(
          choice_batch_size=5, top_n=5, llm=OpenAI(model="grok-2-latest", api_key=XAI_API_KEY, api_base="https://api.x.ai/v1")
      )

I'm also trying to implement this in the synthesize() step and having the same issue. Any guidance would be appreciated!

8 comments

Find answers from the community

Resolving Memory Leaks in Qdrant Vector Database Connections

Modifying the Reranking System in LLMRerank

Integrating Grok Into Llm Reranking System