----------------------
The issue you're facing might be due to the specific setup required for using the
vLLM
model with the
OpenAI
class from LlamaIndex. The
OpenAI
class is designed to work with OpenAI's API, and it might not be directly compatible with the
vLLM
server.
If you're using a
vLLM
server launched with
vllm.entrypoints.openai.api_server
as an OpenAI Compatible Server or via Docker, you need to use the
OpenAILike
class from the
llama-index-llms-openai-like
module. This class is a thin wrapper around the OpenAI model that makes it compatible with 3rd party tools that provide an OpenAI-compatible API.
Here's an example of how to use it:
from llama_index.llms.openai_like import OpenAILike
llm = OpenAILike(model="my model", api_base="<https://hostname.com/v1",> api_key="fake")
Please replace
"my model"
,
"<https://hostname.com/v1"
,> and
"fake"
with your model name, your server's URL, and your API key respectively.
For more details, please refer to the
LlamaIndex documentation ().
----------------------
Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:
----------------------