I am having a issue trying to use Open-

At a glance

A community member is having issues using the Open-Orca/OpenOrca-Platypus2-13B model, with the model chatting with itself and producing unwanted "[/INST]" tags. They found that using the "stop" command in the API helped, but they are unsure if there is a way to do this in llamaindex as well. Other community members suggest setting the "stop" parameter in the "additional_kwargs" of the openai-like constructor, and potentially setting up "messages_to_prompt" and "completion_to_prompt" hooks to properly format the requests. However, the community member could not find these options in the llamaindex documentation or code. Eventually, they were able to solve the issue by adding the "stop" parameter to the kwargs.

Useful resources

iicsy7867

I am having a issue trying to use Open-Orca/OpenOrca-Platypus2-13B. I am gertting [/INST] all over the place and the model keeps chatting with itself. I am using vLLM currently as an "openailike" server.

I looked around the and found an issue where it said to use the STOP command in the API. This made everything work a lot better actually:

Plain Text

curl https://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "Open-Orca/OpenOrca-Platypus2-13B",
        "stop": ["[INST]", "[/INST]"],
        "messages": [
            {"role": "user", "content": "What is the square root of two"}
        ] }'

But I can't see if there is a way for llamaindex to do this as well? I have read through the docs and looked at the code but couldnt figure out if there was an easier way to do this. Any ideas?

9 comments

LLogan M

you can set this under additional_kwargs in the openai like constructor

LLogan M

I don't 100% know if VLLM is doing any prompt templating though

I might recommend also setting up messages_to_prompt and completion_to_prompt hooks to properly format requests using openailike 👀

iicsy7867

Thanks! I checked the docs:
https://docs.llamaindex.ai/en/stable/api_reference/llms/openai_like/

And i did not see the option for messages_to_prompt for openailike

iicsy7867

I also dont see those in the openailike class
https://github.com/run-llama/llama_index/blob/5aa98fa6a94e5406e894a5de30547f61279ff59a/llama-index-integrations/llms/llama-index-llms-openai-like/llama_index/llms/openai_like/base.py#L24

iicsy7867

I will try adding this to the kwargs too.

iicsy7867

The stop in the kwarg seems to have almost solved it

iicsy7867

except now I have a [/SYS] at the end? so odd.

LLogan M

They are kwargs in the parent class init.

You can see them being used in a few places in that file, like here
https://github.com/run-llama/llama_index/blob/5aa98fa6a94e5406e894a5de30547f61279ff59a/llama-index-integrations/llms/llama-index-llms-openai-like/llama_index/llms/openai_like/base.py#L115

iicsy7867

Ah that's awesome. Thank you

Add a reply

Find answers from the community

I am having a issue trying to use Open-