Find answers from the community

Updated 5 months ago

Token usage

At a glance

The community member is asking how to pass certain values (total LLM token usage and total embedding token usage) to a response object along with the answer. Another community member suggests checking the last_token_usage of the embed model or LLM predictor, and the original poster responds with a thank you.

Hey Team, how can I pass these values to a response object together with answer:
INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1221 tokens
INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
L
S
2 comments
I think you'll have to check the last_token_usage of the embed model/llm predictor

index.service_context.llm_predictor.last_token_usage (or embed_model too)
πŸ™ thanks
Add a reply
Sign up and join the conversation on Discord