Find answers from the community

Updated 2 months ago

Embeddings

Thats kinda... Surprising? Embeddings models are really small
L
A
17 comments
That is kind of surprising. It's just a light wrapper around sentence transformers too
I'm assuming that's the class you used
I am trying to train jinaai/jina-embeddings-v2-base-en which is large for an embeddings model
Plain Text
from llama_index.finetuning import (
    generate_qa_embedding_pairs,
    EmbeddingQAFinetuneDataset,
)
from llama_index.finetuning import SentenceTransformersFinetuneEngine
import os
os.environ['HF_TOKEN'] = "***"
train_dataset = EmbeddingQAFinetuneDataset.from_json("data/li_full.json")
val_dataset = EmbeddingQAFinetuneDataset.from_json("data/li_test.json")
#Run Embedding Finetuning
from llama_index.finetuning import SentenceTransformersFinetuneEngine
finetune_engine = SentenceTransformersFinetuneEngine(
    train_dataset,
    model_id="jinaai/jina-embeddings-v2-base-en",
    model_output_path="test_llama_index_finetune",
    val_dataset=val_dataset,
)
finetune_engine.finetune()
embed_model = finetune_engine.get_finetuned_model()
embed_model
Yea it's only like 275 MB hey?

I honestly have no idea what the issue is tbh lol especially since llama index isn't really doing much if you look at the code
I feel like there should be an option to specify batch size lol but it's not there
Actually I think its the context length
Its 8K, and no flash attention or anything like we use in llm land
Shouldn't the context length be reflected in the model weights size though? Or maybe I don't know how context length works anymore in these new models lol

https://huggingface.co/jinaai/jina-embeddings-v2-base-en/tree/main
Nah it is a seperate parameter
In fact, in transformers models it scales non linearly. It scales with model weights as wll, but... TBH I'm not sure why its so hungry.
No flash attn at least
I just thought there had to be trained/saved parameters associated with each token/context position πŸ€” (which should show up in the saved model size?) But yea, otherwise large context size does increae memory usage a ton
Actually this happens with PEFT training as well.
IDK if that's what you meant
Add a reply
Sign up and join the conversation on Discord