Find answers from the community

Updated last month

Can I Define a Stop Keyword with OpenAILike?

At a glance

I am using VLLM to serve up a model. I need to define a keyword to stop. When using OpenAILike, is there a way to define this?

llm=OpenAILike(model=vllm_model, api_base="http://192.168.0.110:8000/v1", api_key="fake",max_tokens=1024)

4 comments

LLogan M

Depends on the API you are wrapping I guess? But its probably additional_kwargs={"some key", "some val"} ?

LLogan M

What that key/val is depends on your API

SSky Candy 🍬🍭

Thank you! I will check the docs for Vllm and give this a try

SSky Candy 🍬🍭

That worked! If anyone else is looking, this is what I used for vllm:

llm=OpenAILike(model=vllm_model, api_base="http://192.168.0.110:8000/v1", api_key="fake",max_tokens=1024,additional_kwargs={"stop":["user:","assistant:"]})

Add a reply