I built a script to insert chunks in my vector store wi...

ttatanfort

I built a script to insert chunks in my vector store with llama index.
I would like to use an external API for the embedding. How to do that in Servicecontext ?

8 comments

kkapa.ai

Please stand by @tatanfort as I search the knowledge base. I normally take about 20 seconds or so. :wave:

kkapa.ai

----------------------
To use an external API for the embedding in LlamaIndex, you would need to create an instance of the embedding model and set it in the ServiceContext object. This is typically done using the .from_defaults() method of ServiceContext, where you can pass your embedding model as an argument.

Here is an example of how to do this using the OpenAIEmbedding model:

Plain Text

from llama_index import ServiceContext, VectorStoreIndex, SimpleDirectoryReader
from llama_index.embeddings import OpenAIEmbedding

embed_model = OpenAIEmbedding()  # replace this with your external API embedding model
service_context = ServiceContext.from_defaults(embed_model=embed_model)

# Optionally set a global service context to avoid passing it into other objects every time
from llama_index import set_global_service_context

set_global_service_context(service_context)

documents = SimpleDirectoryReader("./data").load_data()

index = VectorStoreIndex.from_documents(documents)

In this example, OpenAIEmbedding() is used as the embedding model. You would replace this with your external API embedding model. The ServiceContext object is then created with this embedding model as the default. This context is then used when creating the VectorStoreIndex from documents.

Please note that you would need to ensure that your external API embedding model is compatible with LlamaIndex and follows the same interface as the other embedding models.

For more details, you can refer to the LlamaIndex documentation ().
----------------------
Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:
----------------------

ttatanfort

@kapa.ai I mean the model I use is from hugging face but it's deployed on a VM using a REST API. There is an endpoint to request with takes a list of strings as input and returns a list of embeddings as output.
How to integrate this embedding API with llama index ?

kkapa.ai

Let me check the knowledge base for you. One moment! :books:

kkapa.ai

----------------------
To integrate an external Hugging Face model deployed on a VM with a REST API into LlamaIndex, you would need to create a custom embedding class that interacts with your API. This class should inherit from the base Embedding class in LlamaIndex and implement the get_text_embedding method to make a request to your API and process the response.

Here is a basic example of how you might do this:

Plain Text

import requests
from llama_index.embeddings import Embedding

class MyCustomEmbedding(Embedding):
    def __init__(self, api_url, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.api_url = api_url

    def get_text_embedding(self, text):
        response = requests.post(self.api_url, json={"text": text})
        response.raise_for_status()
        return response.json()["embedding"]

In this example, MyCustomEmbedding is a new class that inherits from Embedding. It takes an api_url as an argument, which is the URL of your API. The get_text_embedding method makes a POST request to your API with the text to be embedded and returns the embedding from the response.

You would then use an instance of MyCustomEmbedding as the embedding model when creating your ServiceContext:

Plain Text

from llama_index import ServiceContext

embed_model = MyCustomEmbedding("<http://my-api-url">)
service_context = ServiceContext.from_defaults(embed_model=embed_model)

Please note that this is a basic example and you might need to adjust it based on the specifics of your API, such as the request format and the response structure.

For more details, you can refer to the LlamaIndex documentation ().
----------------------
Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:
----------------------

ssysfor

@tatanfort did this work for you? Curious if you get it working if you could share

ttatanfort

@sysfor I will try it out. I also need to deep dive the llama index source code a little bit. I'm surprise there is nothing to directly import some class like from llama_index.embeddings import CustomAPIEmbedding where you could just set an endpoint... Maybe this should be a good feature request... (cc: @Logan M what do you think about it?)

LLogan M

Every API is different 🙂 There is an example here of a competely custom embedding class
https://docs.llamaindex.ai/en/stable/examples/embeddings/custom_embeddings.html

If your API is openai-based, you can just use OpenAIEmbedding and change the api_base and model_name kwargs

Add a reply

Find answers from the community

I built a script to insert chunks in my vector store with llama index. I would like to us