Error When Using Vllm

At a glance

The post describes an issue with the VllmServer class from the llama_index.llms.vllm module, where the community member is encountering a KeyError: 'text' error when trying to use the complete() method. In the comments, another community member suggests using the OpenAILike class instead, which allows launching the vLLM in OpenAI API mode. However, there is no explicitly marked answer to the issue described in the post.

rranjanj4

anyone using vllm, it alwsys thros error

Plain Text

 from llama_index.llms.vllm import VllmServer
llm = VllmServer(
    api_url="http://localhost:8000/v1", max_new_tokens=100, temperature=0, device=auto
)

Traceback (most recent call last):
File "/home/jovyan/gen-ai-tm500-llm-nb2-l40s-workspace/TM-500/main_async.py", line 626, in <module>
main()
File "/home/jovyan/gen-ai-tm500-llm-nb2-l40s-workspace/TM-500/main_async.py", line 526, in main
reformulated_query = llm.complete(formatted_prompt)
File "/home/jovyan/gen-ai-tm500-llm-nb2-l40s-workspace/my_env/lib/python3.10/site-packages/llama_index/core/instrumentation/dispatcher.py", line 260, in wrapper
result = func(*args, kwargs) File "/home/jovyan/gen-ai-tm500-llm-nb2-l40s-workspace/my_env/lib/python3.10/site-packages/llama_index/core/llms/callbacks.py", line 429, in wrapped_llm_predict f_return_val = f(_self, *args, kwargs)
File "/home/jovyan/gen-ai-tm500-llm-nb2-l40s-workspace/my_env/lib/python3.10/site-packages/llama_index/llms/vllm/base.py", line 427, in complete
output = get_response(response)
File "/home/jovyan/gen-ai-tm500-llm-nb2-l40s-workspace/my_env/lib/python3.10/site-packages/llama_index/llms/vllm/utils.py", line 9, in get_response
return data["text"]
KeyError: 'text'

1 comment

LLogan M

Launch vLLM in openai api mode, use OpenAILike

Plain Text

llm = OpenAILike(model="some-model", api_base="...", api_key="fake", is_chat_model=True)

Or, feel free to make a PR to clean up this class too, I havent had time to prioritize

Add a reply

Find answers from the community

Error When Using Vllm