I’m using OpenAILike class with vLLM.

At a glance

The community members are discussing the use of the OpenAILike class with a language model. One community member is experiencing an issue where the output includes the "System:" prefix, even when a system prompt is not provided. Another community member suggests that this may be due to the is_chat_model setting being set to False. The solution proposed is to use a custom messages_to_prompt function to format the input to the language model, which can provide more control over the prompt formatting.

Useful resources

TThomas1234

I’m using OpenAILike class with vLLM.

10 comments

LLogan M

yea, use that messages to prompt hook -- this will basically give you universal control on the LLM input

TThomas1234

Okay thanks !

TThomas1234

I might be using it wrong with the chat engine, but when I’m using the OpenAI like with tout the is chat model, even if the system prompt is empty it will send the message as follow
‘System:
Message body’
Is there a way to get rid of this system: at the beginning of every message ?

TThomas1234

I’m thinking that it’s from the LLM metadata when none is being passed to the OpenAI like class ?

LLogan M

Hmm, will have to check this. That typically happens because is_chat_model is being set to false ?

TThomas1234

Yes ! Even when I’m passing a system prompt after it still does
system: {system prompt}

LLogan M

Right, because it's converting the messages to a prompt
https://github.com/run-llama/llama_index/blob/6c8011c4aba179258ca7a011c38f3e8204f1f723/llama-index-integrations/llms/llama-index-llms-openai/llama_index/llms/openai/base.py#L233

You set is_chat_model=True in openai like?

https://github.com/run-llama/llama_index/blob/6c8011c4aba179258ca7a011c38f3e8204f1f723/llama-index-integrations/llms/llama-index-llms-openai-like/llama_index/llms/openai_like/base.py#L42

TThomas1234

No I set is chat model to false

LLogan M

ah, then that would be why 👀 If want is_chat_model=False, but want more control over the formatting of the LLM prompt, you can set a function hook

Plain Text

# very naive formatting, this is what the default is doing essentially
def messages_to_prompt(messages):
  return "\n".join([str(x) for x in messages])

llm = OpenAILike(...., messages_to_prompt=messages_to_prompt)

TThomas1234

Thanks I’ll try it and I’ll let you know !

Add a reply

Find answers from the community

I’m using OpenAILike class with vLLM.