Find answers from the community

Updated 9 months ago

I’m using an embedding model as a

I’m using an embedding model as a dedicated inference server on hf. Is there way to integrate that into llamas embedding models
W
d
11 comments
sort of im hosting it with a cloud url and iI have a url address
Attachment
image.png
Oh okay, you can try custom embedding class:https://docs.llamaindex.ai/en/stable/examples/embeddings/custom_embeddings.html

define the url, make changes in the following:

Plain Text
    def _get_query_embedding(self, query: str) -> List[float]:
        embeddings = MAKE A REQUEST CALL HERE TO YOUR SERVER
        return embeddings[0]

    def _get_text_embedding(self, text: str) -> List[float]:
        embeddings = self._model.encode([[self._instruction, text]])
        return embeddings[0]

    def _get_text_embeddings(self, texts: List[str]) -> List[List[float]]:
        embeddings = self._model.encode(
            [[self._instruction, text] for text in texts]
oh thank god for the code... i would not been able to figure it out
@WhiteFang_Jr sorry im lost

import requests

API_URL = ""
headers = {
"Accept" : "application/json",
"Content-Type": "application/json"
}

def query(payload):
response = requests.post(API_URL, headers=headers, json=payload)
return response.json()

output = query({
"sentences": [

],
"source_sentence": "That is a happy person",
"parameters": {}
})
thats how its called
Your response.json() returns embeddings right?
You'll have to check what kind of output your hosted model is returning coz from the above method you have to return only the embeddings!

also for your case, code would be something like this:

Plain Text
    def _get_query_embedding(self, query: str) -> List[float]:
        # ADDED ONE SAMPLE HERE
        response = requests.post(API_URL, headers=headers, json=payload)
        embeddings = response.json()
        return embeddings

    def _get_text_embedding(self, text: str) -> List[float]:
        embeddings = self._model.encode([[self._instruction, text]])
        return embeddings[0]

    def _get_text_embeddings(self, texts: List[str]) -> List[List[float]]:
        embeddings = self._model.encode(
            [[self._instruction, text] for text in texts]
ohh okay @WhiteFang_Jr do n't i need to intate as well?
TypeError: Can't instantiate abstract class CustomEmbeddings with abstract method _aget_query_embedding
Add a reply
Sign up and join the conversation on Discord