Find answers from the community

Updated 3 months ago

Attributeerror: 'anthropic' object has no attribute 'get_tokenizer'

At a glance

The community members are discussing an issue with the Anthropic library, where the get_tokenizer method is not available. It seems that this method was never implemented by Anthropic, and they do not provide a tokenizer. The community members reference an example in the LlamaIndex documentation that suggests using this method, which is causing confusion. They also discuss the different versions of the Anthropic library and whether downgrading or updating the library might resolve the issue. The community members conclude that the get_tokenizer method has been removed from the Anthropic library, and they are exploring alternative ways to work with the new API.

Useful resources
@Logan M also AttributeError: 'Anthropic' object has no attribute 'get_tokenizer'
do we have fix for this or do we not use tokziner for anthropic anymore
L
d
17 comments
this method was never implemented πŸ‘€ Anthropic doesn't provide a tokenizer afaik
LOL why does the example say to use it
which example?
that actually works for me
what version of anthropic do you have? I have v0.36.0
do i need to downgrade lol worried about compatible issue with latest claude model
Hmm i guess anthropic removed or renamed that method? Will try updating and see where it went

For now if you need the tokenizer, just downgrade, should be fine
Looks like its removed
they closed they tokens too
The new API is pretty nice though if you use it directly. Allows for counting tools, images, etc.

The API in llama-index is limited to string in, int out though. Updating the class in any case right now
Attachment
image.png
Add a reply
Sign up and join the conversation on Discord