Find answers from the community

Updated 3 months ago

Hello as a newcomer to LlamaIndex I m

Hello, as a newcomer to LlamaIndex, I'm wondering if it's possible to utilize models from HuggingFace locally, similar to Langchain. Could you provide an example if it's feasible?
W
M
L
3 comments
I tried this parameters to load my model
CFG.model_name == 'falcon':
tokenizer = AutoTokenizer.from_pretrained("h2oai/h2ogpt-gm-oasst1-en-2048-falcon-7b-v2")

model = AutoModelForCausalLM.from_pretrained("h2oai/h2ogpt-gm-oasst1-en-2048-falcon-7b-v2",
load_in_8bit=True,
device_map='auto',
torch_dtype=torch.float16,
low_cpu_mem_usage=True,
trust_remote_code=True
)
max_len = 1024
task = "text-generation"
T = 0

I didn't specify the context_window
when I run :
from llama_index import ServiceContext
service_context = ServiceContext.from_defaults(chunk_size=1000, llm_predictor=llm, embed_model=embed_model)

from llama_index import set_global_serivce_context
set_global_serivce_context(service_context)

I get AttributeError: 'NoneType' object has no attribute 'context_window'

I tried to add context_window to falcon parameters but it looks like It doesn't have
TypeError: RWForCausalLM.init() got an unexpected keyword argument 'context_window'
You should specify llm=llm in the service context

I see you created the model and tokenizer, you can also pass those directly into the hughingface llm class

Take a look at the docs that were linked above
Add a reply
Sign up and join the conversation on Discord