Hey guys I am trying to count total

At a glance

The community member is trying to count the total tokens used by an agent, including completion, prompt, and embedding, but the sum of total_llm_token_count and total_embedding_token_count does not provide an accurate outcome. Another community member asks for more details and the version of the llama index being used. The second community member shares a function to get a token counter, but the printed token count is often single digits or even 0, even with long prompts. The community member is using llama index version 0.7.12.

TTungdepzai

Hey guys, I am trying to count total token used by an agent, completion, prompt, embedding,.... Using total_llm_token_count + total_embedding_token_count doesnt give accurate outcome.

2 comments

LLogan M

Can you explain a little more? What version of llama index do you have?

TTungdepzai

so I have this function:

def get_token_counter(session):
    token_counter = TokenCountingHandler(
        tokenizer=tiktoken.encoding_for_model(session.model).encode)
    # token_counter.reset_counts()
    callback_manager = CallbackManager([token_counter])

    # set the global service_context
    embed_model = OpenAIEmbedding(embed_batch_size=100)  # ada-002
    llm = OpenAI(model=session.model, temperature=0)
    service_context = ServiceContext.from_defaults(
        embed_model=embed_model, llm=llm, callback_manager=callback_manager)
    set_global_service_context(service_context)
    return token_counter

I get the token_counter then chat with the agent:

token_counter = get_token_counter(session)
.......
 print("TOKEN USEDDDD: ", (token_counter.total_llm_token_count +
              token_counter.total_embedding_token_count))

the print would only be single digits, sometimes even 0, even if I give really long prompts
I have the 0.7.12 llama index installed

Add a reply

Find answers from the community

Hey guys I am trying to count total