Reranker not working with deepseek model

At a glance

The post discusses an issue with the DeepSeek LLM model not working properly with the llama_index library's reranker functionality. Community members suggest that the issue may be due to the DeepSeek model's inability to generate proper JSON responses or follow instructions well, compared to other LLMs like OpenAI and Claude. A community member was able to work around the issue by using the OpenAILike library instead of the DeepSeek library. Additionally, they recommend adding retry logic and increasing the timeout to handle non-200 responses from the DeepSeek model.

Useful resources

BBunkley

Plain Text

from llama_index.llms.deepseek import DeepSeek
llm = DeepSeek(model="deepseek-chat", api_key=os.getenv('DEEP_SEEK_API_KEY'))
Settings.llm = llm
Settings.embed_model = OpenAIEmbedding(model=self.embedding)
Settings.node_parser = node_parser
reranker = LLMRerank(top_n=self.similarity_top_k,)
nodes = reranker.postprocess_nodes(nodes, QueryBundle(f"Food items like {item_name}. Seperate item from adjectives and quantities.")

Reranker is not working with deepseek model

13 comments

WWhiteFang_Jr

Maybe its not able create proper JSON while predicting ? Open-source LLMs are not good at this kind of stuff ( smaller ones )

LLogan M

Some LLMs in general are just bad at following instructions yea

BBunkley

@WhiteFang_Jr literally all other modes like 4o-mini, claude 3.5 haiku are working fine. It's just the deep seek model. Shouldn't there be some kind of separate handling for this in the llama_index library ?

BBunkley

@Logan M

WWhiteFang_Jr

The LLM is the culprit here, some LLMs as @Logan M mentioned are not good at instructions that you have to generate the following items in a predefined structure.

But LLMs like OpenAI and claude have these capabilities

BBunkley

so behind the library there is no fancy logic, just some kind of prompt engineering. Thats sad. I though I could use the new deepseek model for my project for cheap.

BBunkley

Anyways I guess open ai wins in real life and real use case

WWhiteFang_Jr

You can add fancy logic, but it all boils down to your LLM capabilities of what it can do or cant

BBunkley

Did some back reading and I was able to get it working using the OpenAILIke library @WhiteFang_Jr @Logan M

Plain Text

if "deepseek" in self.model:
            # self.llm = DeepSeek(model=self.model, api_key=os.getenv('DEEP_SEEK_API_KEY')) -> this is bugged
            self.llm = OpenAILike(model="deepseek-chat", api_base="https://api.deepseek.com/v1", api_key=os.getenv('DEEP_SEEK_API_KEY'), is_chat_model=True)

If anyone is having trouble running reranker from the base

Plain Text

from llama_index.llms.deepseek import DeepSeek

you can use

Plain Text

from llama_index.llms.openai_like import OpenAILike

instead

BBunkley

Sorry for coming back to this again and again, but the error is not that the LLM model returns a not valid json code. its just that the llama index libraries don't have proper handling for when llm request timesout and gives a non 200 non valid json response. You can curb this problem by doing multiple retries on the api call like this

Plain Text

RETRY_COUNT = 3
while RETRY_COUNT > 0:
try:
response = query_engine.query(prompt + item_name)
break
except Exception as e:
print(f"Error querying: {e}")
RETRY_COUNT -= 1
if not response:
raise Exception("LLM returned nothing even after retrying")

I only encountered this with deepseek model, because maybe its new

LLogan M

You can also set timeout=1000 or something else on the llm

LLogan M

To increase the timeout

LLogan M

Hmm I think i messed up by assuming deepseek had a tools api (it might not). That's the main difference between these two, because you set is_function_calling_model=False on the OpenAILike

Add a reply

Find answers from the community

Reranker not working with deepseek model