Find answers from the community

Updated 2 months ago

Hi everyone, I am planning to use

Hi everyone, I am planning to use SentenceTransformersFinetuneEngine to fine-tune a Sentence embeddings model. However I did not find any documantation of the methodology used to do the fine tuning. Is it some sort of LoRa training or is it just adjusting all weights of the model like a classic training ? Thanks !
L
J
r
4 comments
Under the hood, it's using the fit() method provided by sentence transformers

Not 100% sure, but from my understanding this fine-tunes all model weights. Embedding models are usually small, so no need for LoRa
I will look at Sentence Transformer documentation then
the SentenceTransformersFinetuneEngine class in the codebase is used for fine-tuning Sentence Transformers models. It's not a specific type of training like LoRa (Long Range) training. Instead, it's a classic training approach where all the weights of the model are adjusted based on the training data.

The finetune method in this class calls the fit function of the Sentence Transformers model with several parameters including the training data, the number of epochs, the warmup steps, the output path for the model, whether to show a progress bar, the evaluator, and the evaluation steps.

This method doesn't specify which layers of the model are being fine-tuned, so it's safe to assume that all layers of the model are being fine-tuned. This is typical in transfer learning scenarios where a pre-trained model is fine-tuned on a specific task.

Note that the actual implementation of how the fine-tuning is performed would be in the classes that inherit from BaseEmbeddingFinetuneEngine, such as the SentenceTransformersFinetuneEngine class. The BaseEmbeddingFinetuneEngine class itself is an abstract base class and does not provide an implementation for the finetune method.

this is my best guess
cc: @Logan M can keep me honest
Add a reply
Sign up and join the conversation on Discord