import tiktoken

At a glance

The community members are discussing how to calculate token usage for Gemini-based models, such as Gemini-Pro and Gemini-1.5-Flash-Latest. They explore using the tiktoken library for GPT models, but are unsure if it works for Gemini models. Some suggestions include using the AutoTokenizer from Hugging Face, but it's unclear if that's the same tokenizer used by Gemini. The community members also discuss the possibility of using Gemma, but realize it's only for Google open-source models, not Gemini. Ultimately, they find that using Vertex AI's GenerativeModel and a custom tokenizer function may be a solution, but acknowledge it's a "kind of silly" approach.

nnavya1260

import tiktoken
from llama_index.core.callbacks import CallbackManager, TokenCountingHandler
token_counter = TokenCountingHandler(
tokenizer=tiktoken.encoding_for_model("gpt-3.5-turbo").encode
)
I use this to calculate token usage for gpt models.
How do i calculate token usage for gemini based models like gemini-pro or gemini-1.5-flash-latest?

15 comments

LLogan M

no idea lol does gemini provide a tokenizer?

LLogan M

The tokenizer in this case just has to be a callable function, that takes a string and returns a list

nnavya1260

is it possible to use gemma?

LLogan M

could yea, using AutoTokenizer from huggingface

LLogan M

assuming thats the same tokenizer that gemini uses

nnavya1260

oh i think gemma is only for google open source models

nnavya1260

not for gemini models

nnavya1260

having a hard time figuring how to calculate token usage for gemini.. when doing .query

nnavya1260

🥲

LLogan M

gemma or openai token counting will be "close enough" in most cases imo 😅

LLogan M

looks like you can do this if you are using vertex

nnavya1260

Im using gemini from google developer and not vertex.. in that case i have to try using vertex instead

LLogan M

Plain Text

from vertexai.preview.generative_models import GenerativeModel
gemini_pro_model = GenerativeModel("gemini-pro")

def tokenizer(text):
  count = gemini_pro_model.count_tokens("why is sky blue?")
  return [0]*count

...
tokenizer=tokenizer
...

LLogan M

kind of silly

nnavya1260

ikr😅

Add a reply

Find answers from the community

import tiktoken