Hi, I am using a server that is

bbixqu

Hi, I am using a server that is configured with litellm, and I have an SSH tunnel that hits my localhost. For defining an llm I use :

Plain Text

from llama_index.llms import OpenAILike

llm = OpenAILike(api_base="http://localhost:8000",\
                 model="gpt_35_turbo",
                 api_key="sk-xxx")

and it works perfectly. The problem is I can't seem to use OpenAILike to get my embedding model as well. How can I do this?

25 comments

LLogan M

Ah yea, we haven't setup a class specifically for openai-like embeddings

You can use the existing OpenAIEmbedding class though I think

Plain Text

embed_model = OpenAIEmbedding(model_name="my_model", api_base="http://localhost:8000", api_key="fake")

bbixqu

hah, that was easy, it works. Thanks a lot 🙂

LLogan M

Nice!

IIbrahim

hello @Logan M
I hope you are well
I would like to use a proxy of gpt through localhost
I would like to use it for the RAG CLI
the cli works with my openai account perfectly, but i would also like to use a proxy of gpt through localhost
Any advice you can offer me would be greatly appreciated

IIbrahim

I am getting a 404 error :

IIbrahim

sorry that my whole poetry shell, ill send the relevant part

IIbrahim

I am ready to read through any docs pages you advise me to read or follow any advice you can offer 😁

LLogan M

seems like you have it setup correctly (although gpt-4 is not an embedding model)

Seems like you api_base is not quite configured correctly?

IIbrahim

all my others tools use localhost:8080 with /v1, adn one uses without v1
I tried all api_base variations, including 127.0.0.1 and without http
hmm

IIbrahim

for the embedding model il make it gpt-3.5-turbo

IIbrahim

lemme generate a git patch so you can see what ive changed and what im doing inccorectly and getting the 404

IIbrahim

diff --git a/llama-index-cli/llama_index/cli/rag/base.py b/llama-index-cli/llama_index/cli/rag/base.py
index 03b4942a7..022ff95f0 100644
--- a/llama-index-cli/llama_index/cli/rag/base.py
+++ b/llama-index-cli/llama_index/cli/rag/base.py
@@ -5,6 +5,11 @@ from argparse import ArgumentParser
from glob import iglob
from pathlib import Path
from typing import Any, Callable, Dict, Optional, Union, cast
+from llama_index.llms.openai_like import OpenAILike
+# from llama_index.core.base.embeddings.openai import OpenAIEmbedding
+from llama_index.embeddings.openai import OpenAIEmbedding
+
+

from llama_index.core import (
SimpleDirectoryReader,
@@ -33,7 +38,8 @@ def _try_load_openai_llm():
try:
from llama_index.llms.openai import OpenAI # pants: no-infer-dep

return OpenAI(model="gpt-3.5-turbo", streaming=True)

+ return OpenAILike(api_base="http://localhost:8080/v1", model="gpt-4", api_key="nothing", is_local = True, context_window = 8192)
+ # return OpenAI(model="gpt-3.5-turbo", streaming=True)
except ImportError:
raise ImportError(
"llama-index-llms-openai package not found, "
@@ -120,7 +126,7 @@ class RagCLI(BaseModel):
llm = cast(LLM, values["llm"])

# get embed_model from transformations if possible

embed_model = None

+ embed_model = OpenAIEmbedding(model_name="gpt-3.5-turbo", api_base="http://localhost:8080/v1", api_key="nothing", is_local=True, context_window = 8192)
if ingestion_pipeline.transformations is not None:
for transformation in ingestion_pipeline.transformations:
if isinstance(transformation, BaseEmbedding):

IIbrahim

let me format that right

IIbrahim

Plain Text

diff --git a/llama-index-cli/llama_index/cli/rag/base.py b/llama-index-cli/llama_index/cli/rag/base.py
index 03b4942a7..022ff95f0 100644
--- a/llama-index-cli/llama_index/cli/rag/base.py
+++ b/llama-index-cli/llama_index/cli/rag/base.py
@@ -5,6 +5,11 @@ from argparse import ArgumentParser
 from glob import iglob
 from pathlib import Path
 from typing import Any, Callable, Dict, Optional, Union, cast
+from llama_index.llms.openai_like import OpenAILike
+# from llama_index.core.base.embeddings.openai import OpenAIEmbedding
+from llama_index.embeddings.openai import OpenAIEmbedding
+
+
 
 from llama_index.core import (
     SimpleDirectoryReader,
@@ -33,7 +38,8 @@ def _try_load_openai_llm():
     try:
         from llama_index.llms.openai import OpenAI  # pants: no-infer-dep
 
-        return OpenAI(model="gpt-3.5-turbo", streaming=True)
+        return OpenAILike(api_base="http://localhost:8080/v1", model="gpt-4", api_key="nothing", is_local = True, context_window = 8192)
+        # return OpenAI(model="gpt-3.5-turbo", streaming=True)
     except ImportError:
         raise ImportError(
             "`llama-index-llms-openai` package not found, "
@@ -120,7 +126,7 @@ class RagCLI(BaseModel):
         llm = cast(LLM, values["llm"])
 
         # get embed_model from transformations if possible
-        embed_model = None
+        embed_model = OpenAIEmbedding(model_name="gpt-3.5-turbo", api_base="http://localhost:8080/v1", api_key="nothing", is_local=True, context_window = 8192)
         if ingestion_pipeline.transformations is not None:
             for transformation in ingestion_pipeline.transformations:
                 if isinstance(transformation, BaseEmbedding):

IIbrahim

btw Logan I sent youa connect request with a thank you note on LinkedIn 😁
Nice experience, it's impressive

IIbrahim

imma swithc openailike for openai rq

that...

did it?

let me check

@Logan M
alright thank you very much Logan for your advie, with your help i noticed i was using openalike instead of openai as you suggested in your earlier assistance for another llama_index user, now ican use this tool more
Tahnk you very much 😁

IIbrahim

id like to clarify one thing though

IIbrahim

I'm not getting any errors when i use model='gpt-4' in openaiembeeddings function with my local api, is it advisable moving forward that i keep it as gpt-4 as long as it works?

LLogan M

I think as long as it works, go for it!

LLogan M

Glad it works now though 💪💪

Add a reply

Find answers from the community

Hi, I am using a server that is