Find answers from the community

Updated 2 weeks ago

Error When Using Vllm

anyone using vllm, it alwsys thros error
Plain Text
 from llama_index.llms.vllm import VllmServer
llm = VllmServer(
    api_url="http://localhost:8000/v1", max_new_tokens=100, temperature=0, device=auto
)

Traceback (most recent call last):
File "/home/jovyan/gen-ai-tm500-llm-nb2-l40s-workspace/TM-500/main_async.py", line 626, in <module>
main()
File "/home/jovyan/gen-ai-tm500-llm-nb2-l40s-workspace/TM-500/main_async.py", line 526, in main
reformulated_query = llm.complete(formatted_prompt)
File "/home/jovyan/gen-ai-tm500-llm-nb2-l40s-workspace/my_env/lib/python3.10/site-packages/llama_index/core/instrumentation/dispatcher.py", line 260, in wrapper
result = func(*args, kwargs) File "/home/jovyan/gen-ai-tm500-llm-nb2-l40s-workspace/my_env/lib/python3.10/site-packages/llama_index/core/llms/callbacks.py", line 429, in wrapped_llm_predict f_return_val = f(_self, *args, kwargs)
File "/home/jovyan/gen-ai-tm500-llm-nb2-l40s-workspace/my_env/lib/python3.10/site-packages/llama_index/llms/vllm/base.py", line 427, in complete
output = get_response(response)
File "/home/jovyan/gen-ai-tm500-llm-nb2-l40s-workspace/my_env/lib/python3.10/site-packages/llama_index/llms/vllm/utils.py", line 9, in get_response
return data["text"]
KeyError: 'text'
L
1 comment
Launch vLLM in openai api mode, use OpenAILike

Plain Text
llm = OpenAILike(model="some-model", api_base="...", api_key="fake", is_chat_model=True)


Or, feel free to make a PR to clean up this class too, I havent had time to prioritize
Add a reply
Sign up and join the conversation on Discord