Prompt

At a glance

The community member is working on a project that compares content across multiple documents using an index per document set. They noticed that the queries sent to the local LLM include example questions, which seem to be causing issues. The community members discuss this problem, noting that the example questions are being overly weighted and resulting in excessive, poorly targeted LLM calls that are confusing.

Another issue arises when the community member generates unique tools/indexes based on the CV PDF name. When a query crosses indexes, the llama-index creates a non-existent tool name to search against, causing the code to fail. The community members discuss this issue, with one noting that the LLM is "hallucinating" the tool name, which is a common behavior for open-source LLMs, especially when the tool names are long or lack meaning.

The community members suggest that moving to a stronger (13B) model has resolved the issue.

jjonm

Potential Bug?

I've been working on something that should compare content across multiple documents (i.e an index per document set). When I trace the queries that are being sent to the local LLM, there are some some example questions also passed along with the prompt:

12 comments

LLogan M

This is part of the few shot prompt sent to the llm, showing examples of how the react agent should be working

jjonm

I understand this Logan. Unfortunately, it looks like their importance is being overly weighted and resulting in excessive poorly targeted LLM calls that are confusing.

LLogan M

Yea saw your gh issue. Its a fair point, especially for less capable LLMs

You can already customize the prompt too though

jjonm

I'm just looking at that now in the documentation.

jjonm

However, it looks like I may have stumbled over another issue while showing the prompte being generated

jjonm

I generate unique tools/indexes based on the CV pdf name. However, it looks like when there is a query that crosses indexes (reasonable), that llama-index creates a nonexistent tool name to search against, and the code fails

jjonm

Plain Text

{
    "items": [
        {
            "sub_question": "What technical skills are listed on John Doe's CV (CVs_CV_John_Doe_Candidate_180)",
            "tool_name": "CVs_CV_John_Doe_Candidate_180"
        },
        {
            "sub_question": "What technical skills are listed on Anna Mustermann's CV (CVs_CV_Anna_Mustermann_Candidate_181)",
            "tool_name": "CVs_CV_Anna_Mustermann_Candidate_181"
        },
        {
            "sub_question": "What technical skills are common between John Doe's CV and Anna Mustermann's CV?",
            "tool_name": "CVs_CV_John_Doe_Candidate_180, CVs_CV_Anna_Mustermann_Candidate_181"
        },
        {
            "sub_question": "What unique technical skill does John Doe have that is not listed on Anna Mustermann's CV?",
            "tool_name": "CVs_CV_John_Doe_Candidate_180, CVs_CV_Anna_Mustermann_Candidate_181"
        },
        {
            "sub_question": "What unique technical skill does Anna Mustermann have that is not listed on John Doe's CV?",
            "tool_name": "CVs_CV_John_Doe_Candidate_180, CVs_CV_Anna_Mustermann_Candidate_181"
        }
    ]
}

jjonm

) as event:
233 question = sub_q.sub_question
--> 234 query_engine = self._query_engines[sub_q.tool_name]
236 if self._verbose:
237 print_text(f"[{sub_q.tool_name}] Q: {question}\n", color=color)

KeyError: 'CVs_CV_John_Doe_Candidate_180, CVs_CV_Anna_Mustermann_Candidate_181'

jjonm

I'm not sure if this is a bug in my code, or this is an unexpected generation of a tool name

LLogan M

Llama index isn't creating a tool, the LLM is hallucinating one 😅

This is pretty common behavior for open source LLMs. Especially when the tool names are kind of long/lack meaning like this

LLogan M

The LLM has to write both which tool to use and what the input is

jjonm

Looks like moving to a stronger (i.e. 13B) model has resolved this.

Add a reply

Find answers from the community

Prompt