Found out that it oftentimes produces

ppaulo

Found out that it oftentimes produces the correct JSON but it keeps printing this as part of the response: The new context does not provide any additional information, so the original answer remains the same. so it messes it up when I try loading the response into a JSON file. Do you know how to remove this additional commentary it keeps returning?

17 comments

LLogan M

lol classic gpt-3.5 I'm guessing. Did you customize the refine prompt as well? I forget if you did

ppaulo

I don't think so? I'm passing in my detailed prompt and I'm using QASummaryGraphBuilder

ppaulo

Yes correct gpt-3.5 haha

LLogan M

Yea, you might want to try customizing the refine template as well

I think I have an example somewhere that helped that issue, one sec

LLogan M

Plain Text

from langchain.prompts.chat import (
    AIMessagePromptTemplate,
    ChatPromptTemplate,
    HumanMessagePromptTemplate,
)

from llama_index.prompts.prompts import RefinePrompt

# Refine Prompt
CHAT_REFINE_PROMPT_TMPL_MSGS = [
    HumanMessagePromptTemplate.from_template("{query_str}"),
    AIMessagePromptTemplate.from_template("{existing_answer}"),
    HumanMessagePromptTemplate.from_template(
        "I have more context below which can be used "
        "(only if needed) to update your previous answer.\n"
        "------------\n"
        "{context_msg}\n"
        "------------\n"
        "Given the new context, update the previous answer to better "
        "answer my previous query."
        "If the previous answer remains the same, repeat it verbatim. "
        "Never reference the new context or my previous query directly.",
    ),
]


CHAT_REFINE_PROMPT_LC = ChatPromptTemplate.from_messages(CHAT_REFINE_PROMPT_TMPL_MSGS)
CHAT_REFINE_PROMPT = RefinePrompt.from_langchain_prompt(CHAT_REFINE_PROMPT_LC)
...
query_engine = index.as_query_engine(..., refine_template=CHAT_REFINE_PROMPT)

LLogan M

It wasn't perfect, but seemed to help. Maybe use that as an example lol

EEnderEnder

Wonder if the new guardrails might be something to help with this? Could avoid the extra lengthly prompt template.

ppaulo

Where would I place the refine_template if I'm using the graph with the query_configs?

LLogan M

I think in the query_kwargs of each query config

ppaulo

And I would still pass in the query_str into the query correct?

So something like this:

Plain Text

    # Set query config
    query_configs = [
        {
            "index_struct_type": "simple_dict",
            "query_mode": "default",
            "query_kwargs": {
                "similarity_top_k": 1,
                "response_mode": "compact",
                "refine_template": CHAT_REFINE_PROMPT
            },
        },
        {
            "index_struct_type": "list",
            "query_mode": "default",
            "query_kwargs": {
                "response_mode": "tree_summarize",
                "use_async": True,
                "verbose": True,
                "refine_template": CHAT_REFINE_PROMPT
            },
        },
        {
            "index_struct_type": "tree",
            "query_mode": "default",
            "query_kwargs": {
                "verbose": True,
                "refine_template": CHAT_REFINE_PROMPT
            },
        },
    ]

    try:
        print("BEFORE Querying Graph.query: ", graph)
        response = graph.query(
            query_str=query_str, 
            query_configs=query_configs, 
            service_context=service_context_chatgpt,
        )
...

ppaulo

Can I define the CHAT_REFINE_PROMPT_TMPL_MSGS globally or do I have to define my query_str first and then define CHAT_REFINE_PROMPT_TMPL_MSGS for each separate query?

LLogan M

Nah its global. The format variables get filled in during runtime

ppaulo

When I try running it it just says

The new context does not provide any additional information that would change the previous answer. The previous answer remains the same.

LLogan M

Lol rip. Guess it still doesn't want to follow instructions 🙃

LLogan M

So frustrating to work with gpt3.5... they've really dumbed it down

LLogan M

Assuming it was using the new template I suppose

ppaulo

Yeah I had the same impression, it was working fine a few weeks ago. I'll give guardrails a shot and see if that works

Add a reply

Find answers from the community

Found out that it oftentimes produces