LlamaIndex

Log inLog into community

Find answers from the community

Updated 11 months ago

Hi how to use hosted llamacpp server in

Hi how to use hosted llamacpp server in

At a glance

The community members are discussing how to use a hosted llamacpp server in the llamaindex library for a chat engine. They are trying to use the OpenAILike class, but are encountering an error. The community members have provided code examples and tried various troubleshooting steps, such as updating readers, creating a new environment, and downgrading packages. Eventually, one community member was able to resolve the issue by setting up a new environment and isolating the llamacpp server installation. The community members also discuss the possibility of using the hosted OpenAILike LLM directly in the chat engine.

Useful resources

·

Hi how to use hosted llamacpp server in llamaindex for chat engine. Iam trying by importing OpenAiLike and passing model name and base_api still getting error

L

T

30 comments

What is the error?

Some BaseComponent error. Tired and stopped today. Any alternative apart from using OpenAiLike ? I want to use my base_api and use it for llm calls

from llama_index.llms.openai_like import OpenAILike

llm = OpenAILike(model="my model" api_base="https://hostname.com/v1", api_key="fake")

response = llm.complete("Hello World!")
print(str(response))

I replaced hostname.com with my IP and port.

I would need the full error to help debug 🙂

OpenAILike works fine on my end

Getting validation error
Validation error for data source
Component _type subclass of BaseComponent expected

That seems like an env issue to me. Do you have the exact traceback?

You probably just need to update a reader

Iam just importing one line of code
From llama_index.llms.openai_like import OpenAILike

Then in 2nd line llm =OpenAILike(required params)

I know -- this is related to some other package you have installed

If you give me the exact traceback and I can suggest a fix

Iam behind proxy. I can send one small snip

Attachment

rn_image_picker_lib_temp_c90f17c8-ee2a-4d97-b25b-cf1032ce94ca.jpg

Try updating the elastic search reader pip install -U llama-index-readers-elasticsearch

Or just uninstall it if you aren't using it

It was not installed. Now I installed and upgrades still same issue

Removed url for obvious reasons

Not sure. I can replicate. Looks like you are in a notebook though? Try restarting the runtime/notebook

Done refreshing kernel, created virtual environment. Using only 2 lines of code but giving error.

Don't know what to tell ya 🤷‍♂️ works fine on a fresh Google colab

Maybe try downgrading llama-index-core? Don't use a notebook? It's something cooked in you venv

Well, if you are ever tired of dealing with an issue, I should just fix it myself I guess 😉
https://github.com/run-llama/llama_index/pull/12882

Can u post your colab code. And are u using llamacpp server api ?

not using llamacpp, but thats not related

https://colab.research.google.com/drive/17IJlvx2M2iGu3weIttvwml2axAAt0Vk9?usp=sharing

It's working now. I setup new environment and installed everything new . In my previous environment I have installed llamacpp server too , I don't know if that caused any issue. Now I isolated both environments and it's working. Thanks Logan. I troubled you

Can we define llm.baseapi and using it in chat engine directly ?

not sure what you mean. You can give the api_base to the openailike llm, and then use it wherever

I want my hosted OpenAiLike llm to be used for chat engine also. I will try once

Add a reply

Sign up and join the conversation on Discord