Error When Using Vllm

anyone using vllm, it alwsys thros error

Plain Text

 from llama_index.llms.vllm import VllmServer
llm = VllmServer(
    api_url="http://localhost:8000/v1", max_new_tokens=100, temperature=0, device=auto
)

Traceback (most recent call last):
File "/home/jovyan/gen-ai-tm500-llm-nb2-l40s-workspace/TM-500/main_async.py", line 626, in <module>
main()
File "/home/jovyan/gen-ai-tm500-llm-nb2-l40s-workspace/TM-500/main_async.py", line 526, in main
reformulated_query = llm.complete(formatted_prompt)
File "/home/jovyan/gen-ai-tm500-llm-nb2-l40s-workspace/my_env/lib/python3.10/site-packages/llama_index/core/instrumentation/dispatcher.py", line 260, in wrapper
result = func(*args, kwargs) File "/home/jovyan/gen-ai-tm500-llm-nb2-l40s-workspace/my_env/lib/python3.10/site-packages/llama_index/core/llms/callbacks.py", line 429, in wrapped_llm_predict f_return_val = f(_self, *args, kwargs)
File "/home/jovyan/gen-ai-tm500-llm-nb2-l40s-workspace/my_env/lib/python3.10/site-packages/llama_index/llms/vllm/base.py", line 427, in complete
output = get_response(response)
File "/home/jovyan/gen-ai-tm500-llm-nb2-l40s-workspace/my_env/lib/python3.10/site-packages/llama_index/llms/vllm/utils.py", line 9, in get_response
return data["text"]
KeyError: 'text'

Find answers from the community

Error When Using Vllm