Find answers from the community

Updated last year

ValueError: Tokenizer class LLaMATokeniz...

Currently using Llama2 via HuggingFace.

Encountering this issue:

Plain Text

ValueError: Tokenizer class LLaMATokenizer does not exist or is not currently imported.

Tracked the huggingface error to this: https://github.com/huggingface/transformers/issues/22222#issuecomment-1477171703

but not sure how to fix that llamaindex-wise.

6 comments

LLogan M

Hmmm, I thiiiink you can load the tokenizer outside of llama-index first, and pass it in

Sounds like in the github issue the captalization is causeing issues lol

Plain Text

tokenizer = LlamaTokenizer.from_pretrained("name")

llm = HuggingFaceLLM(..., tokenizer=tokenizer)

pparagoniq

will def try.

pparagoniq

didn't work 😦

LLogan M

really? Which line failed, the top one?

LLogan M

I think you just need to find another huggingface repo that has the config created properly 😆 Last resort is editing the downloaded config files?

pparagoniq

that was it, that model was outdated apparently. sorry for wild goose chase.

I do have a different issues now though 😀 but will start new message for that

as always, thanks!

Add a reply