Find answers from the community

Updated 2 months ago

VLLM Configuration Issues in Llama Index

pavanmantha · 2024-10-17T10:55:22.741Z

I noticed couple of things.vLLM needs the model parameter in the payload but i don't see that is being sent from llama-index-llms-vllmwhy the payload to the /chat/completions is being translated to prompt infact it should be messages: [{role:'',content:''},{role:'',content:''}]

ppavanmantha

I noticed couple of things.

vLLM needs the model parameter in the payload but i don't see that is being sent from llama-index-llms-vllm
why the payload to the /chat/completions is being translated to prompt infact it should be messages: [{role:'',content:''},{role:'',content:''}]

5 comments

WWhiteFang_Jr

For second,
I think the structure of the VLLM server class was likely created that way when it was first added.

ppavanmantha

So this needs modification then

WWhiteFang_Jr

Yes definitely

ppavanmantha

the reason for this is you can not use /chat/completions, we should use /completions because the payload is not compatable with openai format

ppavanmantha

i hope i am making it clear

Add a reply