Find answers from the community

Updated 3 months ago

I’m using OpenAILike class with vLLM.

TThomas1234

10 comments

LLogan M

yea, use that messages to prompt hook -- this will basically give you universal control on the LLM input

TThomas1234

Okay thanks !

TThomas1234

I might be using it wrong with the chat engine, but when I’m using the OpenAI like with tout the is chat model, even if the system prompt is empty it will send the message as follow
‘System:
Message body’
Is there a way to get rid of this system: at the beginning of every message ?

TThomas1234

I’m thinking that it’s from the LLM metadata when none is being passed to the OpenAI like class ?

LLogan M

Hmm, will have to check this. That typically happens because is_chat_model is being set to false ?

TThomas1234

Yes ! Even when I’m passing a system prompt after it still does
system: {system prompt}

LLogan M

Right, because it's converting the messages to a prompt
https://github.com/run-llama/llama_index/blob/6c8011c4aba179258ca7a011c38f3e8204f1f723/llama-index-integrations/llms/llama-index-llms-openai/llama_index/llms/openai/base.py#L233

You set is_chat_model=True in openai like?

https://github.com/run-llama/llama_index/blob/6c8011c4aba179258ca7a011c38f3e8204f1f723/llama-index-integrations/llms/llama-index-llms-openai-like/llama_index/llms/openai_like/base.py#L42

TThomas1234

No I set is chat model to false

LLogan M

ah, then that would be why 👀 If want is_chat_model=False, but want more control over the formatting of the LLM prompt, you can set a function hook

Plain Text

# very naive formatting, this is what the default is doing essentially
def messages_to_prompt(messages):
  return "\n".join([str(x) for x in messages])

llm = OpenAILike(...., messages_to_prompt=messages_to_prompt)

TThomas1234

Thanks I’ll try it and I’ll let you know !

Add a reply