ValueError: Tokenizer class LLaMATokeniz...

At a glance

The community member is using Llama2 via HuggingFace and encountering an issue with the tokenizer. They have tracked the issue to a GitHub comment, but are unsure how to fix it. The comments suggest trying to load the tokenizer outside of llama-index and pass it in, as the capitalization may be causing issues. However, this did not work. Another community member suggests finding another HuggingFace repo with the config created properly, or as a last resort, editing the downloaded config files. The original poster confirms that the model was outdated, and they now have a different issue to address.

Useful resources

pparagoniq

Currently using Llama2 via HuggingFace.

Encountering this issue:

Plain Text

ValueError: Tokenizer class LLaMATokenizer does not exist or is not currently imported.

Tracked the huggingface error to this: https://github.com/huggingface/transformers/issues/22222#issuecomment-1477171703

but not sure how to fix that llamaindex-wise.

6 comments

LLogan M

Hmmm, I thiiiink you can load the tokenizer outside of llama-index first, and pass it in

Sounds like in the github issue the captalization is causeing issues lol

Plain Text

tokenizer = LlamaTokenizer.from_pretrained("name")

llm = HuggingFaceLLM(..., tokenizer=tokenizer)

pparagoniq

will def try.

pparagoniq

didn't work 😦

LLogan M

really? Which line failed, the top one?

LLogan M

I think you just need to find another huggingface repo that has the config created properly 😆 Last resort is editing the downloaded config files?

pparagoniq

that was it, that model was outdated apparently. sorry for wild goose chase.

I do have a different issues now though 😀 but will start new message for that

as always, thanks!

Add a reply

Find answers from the community

ValueError: Tokenizer class LLaMATokeniz...