Log in
Log into community
Find answers from the community
View all posts
Related posts
Did this answer your question?
π
π
π
Powered by
Hall
Inactive
Updated 2 months ago
0
Follow
llama_index/llama_index/llms/openai_like...
llama_index/llama_index/llms/openai_like...
Inactive
0
Follow
K
Kia
last year
Β·
Would someone be able to assist here? I am trying to use the OpenAILike class with vLLM and am not sure where it makes this call to vLLM and have tried searching for a while. Would someone be able to assist?
https://github.com/run-llama/llama_index/blob/main/llama_index/llms/openai_like.py
L
K
12 comments
Share
Open in Discord
L
Logan M
last year
It uses the OpenAI class to handle the calling
There's a few entry points, but for example
https://github.com/run-llama/llama_index/blob/1033d30a0d2bdd5119834d72d0b8c283a00acf13/llama_index/llms/openai.py#L346
K
Kia
last year
Thanks @Logan M , but how do you know this is calling vLLM?
K
Kia
last year
I don't see this call
K
Kia
last year
Or can't follow the logic on this
L
Logan M
last year
You have to set the base url to point to your vLLM server
L
Logan M
last year
then it calls
L
Logan M
last year
you can set this to see some slightly more detailed logs from the client
Plain Text
Copy
import os os.environ['OPENAI_LOG'] = "debug"
K
Kia
last year
Thanks
K
Kia
last year
@Logan M however, but there isn't really an explicit call to vllm that I can see?
K
Kia
last year
I'm still not sure where this is happening - can you point me to an example of this
L
Logan M
last year
The openai client calls the vllm server at the provided host url.
Since vllm has the same api as openai, we only need to adjust the base url, and the openai client does the work
Any time self._client is called, it's using the openai client to route requests to vllm
https://github.com/run-llama/llama_index/blob/1033d30a0d2bdd5119834d72d0b8c283a00acf13/llama_index/llms/openai.py#L350
That line exactly is an example of calling vllm
L
Logan M
last year
This is as specific as I can get π
Add a reply
Sign up and join the conversation on Discord
Join on Discord