I’m using an embedding model as a

ddrewskidang

I’m using an embedding model as a dedicated inference server on hf. Is there way to integrate that into llamas embedding models

11 comments

WWhiteFang_Jr

Are you looking for this: https://docs.llamaindex.ai/en/stable/examples/embeddings/text_embedding_inference.html ?

ddrewskidang

sort of im hosting it with a cloud url and iI have a url address

Attachment

WWhiteFang_Jr

Oh okay, you can try custom embedding class:https://docs.llamaindex.ai/en/stable/examples/embeddings/custom_embeddings.html

define the url, make changes in the following:

Plain Text

    def _get_query_embedding(self, query: str) -> List[float]:
        embeddings = MAKE A REQUEST CALL HERE TO YOUR SERVER
        return embeddings[0]

    def _get_text_embedding(self, text: str) -> List[float]:
        embeddings = self._model.encode([[self._instruction, text]])
        return embeddings[0]

    def _get_text_embeddings(self, texts: List[str]) -> List[List[float]]:
        embeddings = self._model.encode(
            [[self._instruction, text] for text in texts]

ddrewskidang

oh thank god for the code... i would not been able to figure it out

ddrewskidang

@WhiteFang_Jr sorry im lost

import requests

API_URL = ""
headers = {
"Accept" : "application/json",
"Content-Type": "application/json"
}

def query(payload):
response = requests.post(API_URL, headers=headers, json=payload)
return response.json()

output = query({
"sentences": [

],
"source_sentence": "That is a happy person",
"parameters": {}
})

ddrewskidang

thats how its called

WWhiteFang_Jr

Your response.json() returns embeddings right?

ddrewskidang

I think so

WWhiteFang_Jr

You'll have to check what kind of output your hosted model is returning coz from the above method you have to return only the embeddings!

also for your case, code would be something like this:

Plain Text

    def _get_query_embedding(self, query: str) -> List[float]:
        # ADDED ONE SAMPLE HERE
        response = requests.post(API_URL, headers=headers, json=payload)
        embeddings = response.json()
        return embeddings

    def _get_text_embedding(self, text: str) -> List[float]:
        embeddings = self._model.encode([[self._instruction, text]])
        return embeddings[0]

    def _get_text_embeddings(self, texts: List[str]) -> List[List[float]]:
        embeddings = self._model.encode(
            [[self._instruction, text] for text in texts]

ddrewskidang

ohh okay @WhiteFang_Jr do n't i need to intate as well?

ddrewskidang

TypeError: Can't instantiate abstract class CustomEmbeddings with abstract method _aget_query_embedding

Add a reply

Find answers from the community

I’m using an embedding model as a