since hf is openai-compatiable can we

ddrewskidang

since hf is openai-compatiable can we swap that with openai model or would have to make a custom model lol. Langchain llms are broken again on their side

24 comments

LLogan M

You can use the OpenAILike class

LLogan M

pip install llama-index-llms-openai-like

Plain Text

from llama_index.llms.openai_like import OpenAILike

llm = OpenAILike(model="some model", api_key="fake", api_base="....")

ddrewskidang

thats a thing?

LLogan M

yessir

ddrewskidang

I spend more time reading llama index than my actual school work

ddrewskidang

LOL thanks g

LLogan M

lol theres a lot in here 😆

ddrewskidang

whats the install?

LLogan M

Its up there, but pip install llama-index-llms-openai-like

ddrewskidang

oh didnt see it sorry

ddrewskidang

@Logan M my do i just my huggingface host or do i have to intalize it like this

from openai import OpenAI

initialize the client but point it to TGI

client = OpenAI(
base_url="<ENDPOINT_URL>" + "/v1/", # replace with your endpoint url
api_key="<HF_API_TOKEN>", # replace with your token
)

im getting
llama_index.llms.openai.base.OpenAI._complete in 0.19009121318146227 seconds as it raised NotFoundError: Not Found.
Retrying llama_index.llms.openai.base.OpenAI._complete in 0.4330981257225379 seconds as it raised NotFoundError: Not Found.
Retrying llama_index.llms.openai.base.OpenAI._complete in 3.315326791265811 seconds as it raised NotFoundError: Not Found.

LLogan M

Plain Text

from llama_index.llms.openai_like import OpenAILike

# Instantiate an OpenAILike model
llm = OpenAILike(
    model="tgi",
    api_key="<HF_API_TOKEN>",
    api_base="<ENDPOINT_URL>" + "/v1/",
    is_chat_model=True,
    is_local=False,
    is_function_calling_model=False,
  context_window=32000,
)

LLogan M

Should be able to do that

LLogan M

(With the proper context window size of course)

ddrewskidang

gotcha gotcha I dont the the right

ddrewskidang

that always thorws me off

LLogan M

Yea you don't need the angle brackets

ddrewskidang

Retrying llama_index.llms.openai.base.OpenAI._chat in 8.648145082827636 seconds as it raised UnprocessableEntityError: Error code: 422 - {'error': 'Template error: template not found', 'error_type': 'template_error'}.

ddrewskidang

better not ther yet

ddrewskidang

WORKS

ddrewskidang

had to set to false lol

ddrewskidang

chat_model

LLogan M

Ah that makes sense

LLogan M

Nice

Add a reply

Find answers from the community

since hf is openai-compatiable can we

initialize the client but point it to TGI