Find answers from the community

Updated 3 months ago

import tiktoken

import tiktoken
from llama_index.core.callbacks import CallbackManager, TokenCountingHandler
token_counter = TokenCountingHandler(
tokenizer=tiktoken.encoding_for_model("gpt-3.5-turbo").encode
)
I use this to calculate token usage for gpt models.
How do i calculate token usage for gemini based models like gemini-pro or gemini-1.5-flash-latest?

15 comments

LLogan M

no idea lol does gemini provide a tokenizer?

LLogan M

The tokenizer in this case just has to be a callable function, that takes a string and returns a list

nnavya1260

is it possible to use gemma?

LLogan M

could yea, using AutoTokenizer from huggingface

LLogan M

assuming thats the same tokenizer that gemini uses

nnavya1260

oh i think gemma is only for google open source models

nnavya1260

not for gemini models

nnavya1260

having a hard time figuring how to calculate token usage for gemini.. when doing .query

nnavya1260

🥲

LLogan M

gemma or openai token counting will be "close enough" in most cases imo 😅

LLogan M

looks like you can do this if you are using vertex

nnavya1260

Im using gemini from google developer and not vertex.. in that case i have to try using vertex instead

LLogan M

Plain Text

from vertexai.preview.generative_models import GenerativeModel
gemini_pro_model = GenerativeModel("gemini-pro")

def tokenizer(text):
  count = gemini_pro_model.count_tokens("why is sky blue?")
  return [0]*count

...
tokenizer=tokenizer
...

LLogan M

kind of silly

nnavya1260

ikr😅

Add a reply