I guess there is Issue with vLLM calling
I have the below simple code
from llama_index.llms.vllm import VllmServer
from llama_index.core.llms import ChatMessage, ChatResponse
llm = VllmServer(
model="meta-llama/Llama-3.1-8B-Instruct",
api_url="https://<YOUR_HOST>/v1/chat/completions")
messages = [
ChatMessage(role="system",
content="You are an expert language translator, always translate the given text to a destnation language. neighter explanation nor additional details are needed. just respond with the translated text."),
ChatMessage(role="user", content="Translate ##I Love NLP## to French")
]
response: ChatResponse = llm.chat(messages=messages)
print(response)
When i run the code i end up with the below Error !!
/Users/pavanmantha/Desktop/machine_translation/venv/bin/python /Users/pavanmantha/Desktop/machine_translation/mt_playground.py
Traceback (most recent call last):
File "/Users/pavanmantha/Desktop/machine_translation/mt_playground.py", line 14, in <module>
response: ChatResponse = llm.chat(messages=messages)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/pavanmantha/Desktop/machine_translation/venv/lib/python3.12/site-packages/llama_index/core/instrumentation/dispatcher.py", line 307, in wrapper
result = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/Users/pavanmantha/Desktop/machine_translation/venv/lib/python3.12/site-packages/llama_index/core/llms/callbacks.py", line 173, in wrapped_llm_chat
f_return_val = f(_self, messages, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/pavanmantha/Desktop/machine_translation/venv/lib/python3.12/site-packages/llama_index/llms/vllm/base.py", line 271, in chat
completion_response = self.complete(prompt, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/pavanmantha/Desktop/machine_translation/venv/lib/python3.12/site-packages/llama_index/core/instrumentation/dispatcher.py", line 307, in wrapper
result = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/Users/pavanmantha/Desktop/machine_translation/venv/lib/python3.12/site-packages/llama_index/core/llms/callbacks.py", line 431, in wrapped_llm_predict
f_return_val = f(_self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/pavanmantha/Desktop/machine_translation/venv/lib/python3.12/site-packages/llama_index/llms/vllm/base.py", line 436, in complete
output = get_response(response)
^^^^^^^^^^^^^^^^^^^^^^
File "/Users/pavanmantha/Desktop/machine_translation/venv/lib/python3.12/site-packages/llama_index/llms/vllm/utils.py", line 9, in get_response
return data["text"]
~~~~^^^^^^^^
KeyError: 'text'
But the same works in my HTTPie.