Find answers from the community

Updated 4 months ago

The example of using the Replicate API

At a glance

The example of using the Replicate API with LLaMa2 uses a completion_to_prompt and messages_to_prompt, but those using a HuggingFaceLLM() seem to use a system_prompt and query_wrapper_prompt. How do I migrate from the former to the latter correctly?

1 comment

LLogan M

In the huggingface LLM, just set the query wrapper prompt. It will look something like this

Plain Text

BOS, EOS = "<s>", "</s>"
B_INST, E_INST = "[INST]", "[/INST]"
B_SYS, E_SYS = "<<SYS>>\n", "\n<</SYS>>\n\n"

query_wrapper_prompt=(
    f"{BOS}{B_INST} {B_SYS}{system_prompt_str.strip()}{E_SYS}"
    f"{completion.strip()} {E_INST}"
)

If you aren't using a system prompt, then it would look like this

Plain Text

query_wrapper_prompt=(
    f"{BOS}{B_INST} "
    f"{completion.strip()} {E_INST}"
)

Tbh we should probably align the interface though

Add a reply