I feel a bit dumb here but my

mmcmancsu

I feel a bit dumb here, but my RefinePrompt doesn't appear to be working.. The prompt doesn't seem to take any context into consideration (for example, it doesn't lead with "I am a ROBOT".

Here is what I have:

Plain Text

DEFAULT_REFINE_PROMPT_TMPL = (
    "The original question is as follows: {query_str}\n"
    "We have provided an existing answer: {existing_answer}\n"
    "We have the opportunity to refine the existing answer "
    "(only if needed) with some more context below.\n"
    "------------\n"
    "{context_msg}\n"
    "------------\n"
    "Given the new context and using the best of your knowledge, improve the existing answer. "
    "If you can't improve the existing answer, just repeat it again. Lastly, prefix all responses "
    "with 'I am a ROBOT!'"
)

    index = GPTSimpleVectorIndex.load_from_disk(output_file_html)

    chat_refine_prompt = RefinePrompt(DEFAULT_REFINE_PROMPT_TMPL)
    response = index.query(query_str=question,
                           response_mode="compact",
                           refine_template=chat_refine_prompt,
                           similarity_top_k=3)

17 comments

mmcmancsu

I am new to a lot of this too, and so I am not sure if the query_str and what not need to be explicitly set on the string with the format () function

LLogan M

Are you using gpt3.5 ?

mmcmancsu

yessir. Index was constructed with

    llm_predictor = LLMPredictor(
        llm=ChatOpenAI(temperature=0.7, model_name="gpt-3.5-turbo", max_tokens=num_outputs))

LLogan M

Perfect! So you might want to create your refine template in a way that uses the chat roles abstraction instead. Check out this example:

Plain Text

from langchain.prompts.chat import (
    AIMessagePromptTemplate,
    ChatPromptTemplate,
    HumanMessagePromptTemplate,
    SystemMessagePromptTemplate
)

from llama_index.prompts.prompts import RefinePrompt

# Refine Prompt
CHAT_REFINE_PROMPT_TMPL_MSGS = [
    HumanMessagePromptTemplate.from_template("{query_str}"),
    AIMessagePromptTemplate.from_template("{existing_answer}"),
    HumanMessagePromptTemplate.from_template(
        "I have more context below which can be used "
        "(only if needed) to update your previous answer.\n"
        "------------\n"
        "{context_msg}\n"
        "------------\n"
        "Given the new context, update the previous answer to better "
        "answer my previous query."
        "If the previous answer remains the same, repeat it verbatim. "
        "Never reference the new context or my previous query directly.",
    ),
]


CHAT_REFINE_PROMPT_LC = ChatPromptTemplate.from_messages(CHAT_REFINE_PROMPT_TMPL_MSGS)
CHAT_REFINE_PROMPT = RefinePrompt.from_langchain_prompt(CHAT_REFINE_PROMPT_LC)
...
index.query("my query", similarity_top_k=3, refine_template=CHAT_REFINE_PROMPT)

mmcmancsu

Great I'll give it a shot. Is there a simple way to show me exactly what is being sent to the API endpoints?

LLogan M

(Also just a note, you also might want to lower the temperature to make the model a little more "tame")

If you want to see exactly what's being sent/received, check out the llama logger (bottom of this notebook)
https://github.com/jerryjliu/llama_index/blob/main/examples/vector_indices/SimpleIndexDemo.ipynb

mmcmancsu

Thanks. Ok, so I updated it with a couple extra instructions to prove it is working (i.e. prefix, and "add bullets to content), but still doesn't sem to use it 😕 . Am I correct that the variables such as query_str are supposed to be replaced in that template automatically, and that the {existing_answer} will automatically keep the history?

mmcmancsu

I am on the last 0.4 release prior to the refactor in case that is part of the reason

LLogan M

Yea those all get filled in automatically by llama index

Tbh I'm not surprised. A week or two ago I noticed OpenAI must have updated chatGPT, it's been very difficult to get it to follow instructions. A major downgrade, at least from what I've seen

That prompt template I shared was my attempt to get the refine process working a little better.

LLogan M

Out of curiosity, are you also setting the text_qa_template?

mmcmancsu

I am not - just the refine

LLogan M

Oh! You'll want to customize that as well, that might be why you are facing difficulties. I'll link what the default one looks like

mmcmancsu

well that would be cool 😄

LLogan M

This is the default text_qa_template for both gpt3.5 and davinci

For gpt3.5, it gets converted to a single human message at some point

https://github.com/jerryjliu/llama_index/blob/727deb4793042259f4f5993c3b656a1f53ff7b04/gpt_index/prompts/default_prompts.py#L105

mmcmancsu

Ah you rock man! Yeah, so adding both helped:

Plain Text

    index = GPTSimpleVectorIndex.load_from_disk(output_file_html)
    CHAT_REFINE_PROMPT_LC = ChatPromptTemplate.from_messages(CHAT_REFINE_PROMPT_TMPL_MSGS)
    CHAT_REFINE_PROMPT = RefinePrompt.from_langchain_prompt(CHAT_REFINE_PROMPT_LC)
    QA_PROMPT = QuestionAnswerPrompt(DEFAULT_TEXT_QA_PROMPT_TMPL)

    response = index.query(query_str=question,
                           response_mode="compact",
                           refine_template=CHAT_REFINE_PROMPT,
                           text_qa_template=QA_PROMPT,
                           similarity_top_k=3)

mmcmancsu

I added instructions to prove it in the qa_template to test:

Plain Text

    "Start each response with 'I am a bot'. \n"
    "Include 1-4 bullet links to the relevant content."

Neat - thasnk!

LLogan M

Amazing! Glad it worked 💪💪

Add a reply

Find answers from the community

I feel a bit dumb here but my