Find answers from the community

Updated 3 months ago

Counting tokens

Finally, for one example query, I counted ~5k tokens for the above prompt chain ($0.01 with turbo). However, the final cost seemed to be in the $0.30-$0.40 range. Any idea what I'm missing from the final token count?

12 comments

LLogan M

5K token usage was reported by the logs?

BBioHacker

I have found similar case where the logs says like 1000 token but then you look at the open ai panel and it’s usually 2X sometimes even 5X higher than the reported token in llama. Not sure if it’s on my end but it seems like @aleks_wordcab is experiencing something similar.

LLogan M

In my experience the openAI dashboard is pretty laggy 🤔 and ends up grouping together some calls.

but I haven't spent too much time trying to cross reference specific queries to charges in my account

aaleks_wordcab

Correct, logs + my own investigation showed ~5k tokens, but charges were 30x higher

LLogan M

What kind of index are you querying?

aaleks_wordcab

simple vector index

LLogan M

Hmm 🤔

What if you check the tokens on the llm_predictor directly?

llm_predictor.total_tokens_used

Or

llm_predictor.last_token_usage

aaleks_wordcab

actually - scratch the above

aaleks_wordcab

was doing the calculation wrong on my side