How to Trim Text to Fit an Embedding Model Without Knowing the Tokenizer or Input Size
How to Trim Text to Fit an Embedding Model Without Knowing the Tokenizer or Input Size
At a glance
The community member is asking how to ensure text fits into an embedding model, as they cannot know the tokenizer or input size of the model a priori. The community members suggest using tiktoken to count the number of tokens in the text, and that most embedding models simply truncate the text if it exceeds the input size. For the NVIDIA model specifically, the community members note that there is a parameter to control the truncation behavior, such as NVIDIAEmbedding(model="nvidia/nv-embedqa-e5-v5", truncate="END"). Another suggestion is to create summaries of larger documents by summarizing chunks with overlaps, using a tool like documentsummaryindex.
hey all: how do you ensure text fits into an embedding model? you can't know apriori what tokenizer an embedding model uses - or even its input size! Or can you somehow? if I have some arbitrary string 'text' and I need to trim it shorter so it fits into 'embed_model', whats the approach?
You can use tiktoken to count the number of tokens in the text given. Another thing I’m doing is creating summaries of larger document where you summarize chunks with overlaps. Check out documentsummaryindex
