Find answers from the community

Updated last year

I m trying to use `meta llama Llama 2

I'm trying to use meta-llama/Llama-2-70b-chat-hf via the llama_index.llms.HuggingFaceLLM abstraction.

I need to authenticate with HuggingFace (via a user token) in order to use that model.

how can I authenticate/pass the auth token in to HugginfFaceLLM ?

7 comments

LLogan M

tbh I have no idea lol

It will either be in the model_kwargs or will be an env variable you have to set?

pparagoniq

I know that it's the hub_token param for the HuggingFace lib

LLogan M

You could just load the model/tokenizer outside of the LLM class if that's easier

Plain Text

model = ...
llm = HuggingFaceLLM(model=model, tokenizer=tokenizer, ...)

LLogan M

otherwise, setting model_kwargs with that token might do it?

LLogan M

llm = HuggingFaceLLM(..., model_kwargs={"hub_token": "..."})

pparagoniq

doesn't work in the kwargs.

not familiar with the approach on the first suggestion, sorry.

mind pointing me to the docs to see where/how I can do that.

thanks.

LLogan M

I'm actually not sure if it's documented lol but I created this LLM class 😆

but basically, load the model/tokenizer out side of the llama-index LLM class, using Llama

Plain Text

from transformers import LlamaForCausalLM, LlamaTokenizer

# I have no idea if this is how you use hub_token, just guessing
tokenizer = LlamaTokenizer.from_pretrained("name", hub_token="..")
model = LlamaForCausalLM.from_pretrained("name", hub_token="..")

llm = HuggingFaceLLM(model=model, tokenizer=tokenizer, ...)

Add a reply