TrNsform

This entire stack is used 👀

The output of the last layer is called the hidden state/layer, and this is what's used for embeddings.

There's an embedding dimension for each token, and these are typically pooled (i.e. using averaging, or other techniques)

is that the default with embeddings models?

It is, at least to my knowledge 👍

does that mean I could simply take the encoder part of an LLM to calculate embeddings?

especially if its fine-tuned on a specific task close to the retrieval task?

Most LLMs are decoder only actually.

But there is ways to get embeddings from decoder models, I know llama.cpp does it (I can't remember how it works exactly though)

I know thats the lingo some people use. However I was merely talking about the part without lm_head so linear layer and softmax to calculate the emebddings, which is always just an encoder 🤓 Because if you take away the head from a decoder, its basically an encoder.