Log in
Log into community
Find answers from the community
View all posts
Related posts
Did this answer your question?
๐
๐
๐
Powered by
Hall
Inactive
Updated 2 months ago
0
Follow
Is there any plans for standardizing the
Is there any plans for standardizing the
Inactive
0
Follow
s
sansmoraxz
9 months ago
ยท
Is there any plans for standardizing the metadata in responses? Picking raw response for example to count the tokens will not scale across the many different providers.
s
L
18 comments
Share
Open in Discord
s
sansmoraxz
9 months ago
I would suggest capturing
input_tokens_count
,
output_tokens_count
s
sansmoraxz
9 months ago
stop_reason
,
stop_sequence
for guided capturing stop-words
s
sansmoraxz
9 months ago
On the same thought, stop words parameter could be part of BaseLLM as there are some deviations in parameter names when passed to some providers.
L
Logan M
9 months ago
Not really any plans beyond what's already there (capturing the text/message)
It would be a large effort. And not every api provides those items in the response
s
sansmoraxz
9 months ago
OK. I was going through the bedrock API docs.
Thought to add token count capturing for whichever were supported. Saw the current implementation and thought an issue may arise.
s
sansmoraxz
9 months ago
Is it OK if I still add those and do a slight refactor?
s
sansmoraxz
9 months ago
Also just so we are clear the
usage
dict capture is for anthropic right?
L
Logan M
9 months ago
which usage dict capture? In the token counter?
s
sansmoraxz
9 months ago
yes
L
Logan M
9 months ago
Mainly meant for openai, plus any other API that returns usage in a similar format
s
sansmoraxz
9 months ago
OK
s
sansmoraxz
9 months ago
For anthropic it's input_tokens and output_tokens
s
sansmoraxz
9 months ago
but in the same dict
s
sansmoraxz
9 months ago
or maybe, we can expose the TokenCounter in the handler and each integration can have their own implementations. wdyt?
s
sansmoraxz
9 months ago
or this. default can be none or nan for whichevers are not provided
L
Logan M
9 months ago
That's not a bad idea tbh
s
sansmoraxz
9 months ago
I gave this some thought. I think the metadata one will be easier to set up, and much more useful long term, viz directed ReAct agent.
s
sansmoraxz
9 months ago
We can have some glue code in the the token counter interim to use the metadata if it is available or else fallback to original logic.
Add a reply
Sign up and join the conversation on Discord
Join on Discord