Can I Define a Stop Keyword with OpenAILike?

Question

I am using VLLM to serve up a model. I need to define a keyword to stop. When using OpenAILike, is there a way to define this?

llm=OpenAILike(model=vllm_model, api_base="http://192.168.0.110:8000/v1", api_key="fake",max_tokens=1024)

Logan M · Answer

Depends on the API you are wrapping I guess? But its probably additional_kwargs={"some key", "some val"} ?

Logan M · Answer

What that key/val is depends on your API

Sky Candy 🍬🍭 · Answer

Thank you! I will check the docs for Vllm and give this a try

Sky Candy 🍬🍭 · Answer

That worked! If anyone else is looking, this is what I used for vllm:

llm=OpenAILike(model=vllm_model, api_base="http://192.168.0.110:8000/v1", api_key="fake",max_tokens=1024,additional_kwargs={"stop":["user:","assistant:"]})

Find answers from the community

Can I Define a Stop Keyword with OpenAILike?