Hey guys I am trying to count total

Can you explain a little more? What version of llama index do you have?

so I have this function:
def get_token_counter(session): token_counter = TokenCountingHandler( tokenizer=tiktoken.encoding_for_model(session.model).encode) # token_counter.reset_counts() callback_manager = CallbackManager([token_counter]) # set the global service_context embed_model = OpenAIEmbedding(embed_batch_size=100) # ada-002 llm = OpenAI(model=session.model, temperature=0) service_context = ServiceContext.from_defaults( embed_model=embed_model, llm=llm, callback_manager=callback_manager) set_global_service_context(service_context) return token_counter
I get the token_counter then chat with the agent:
token_counter = get_token_counter(session) ....... print("TOKEN USEDDDD: ", (token_counter.total_llm_token_count + token_counter.total_embedding_token_count))
the print would only be single digits, sometimes even 0, even if I give really long prompts
I have the 0.7.12 llama index installed