Hello, as a newcomer to LlamaIndex, I'm wondering if it's possible to utilize models from HuggingFace locally, similar to Langchain. Could you provide an example if it's feasible?
I tried this parameters to load my model CFG.model_name == 'falcon': tokenizer = AutoTokenizer.from_pretrained("h2oai/h2ogpt-gm-oasst1-en-2048-falcon-7b-v2")
model = AutoModelForCausalLM.from_pretrained("h2oai/h2ogpt-gm-oasst1-en-2048-falcon-7b-v2", load_in_8bit=True, device_map='auto', torch_dtype=torch.float16, low_cpu_mem_usage=True, trust_remote_code=True ) max_len = 1024 task = "text-generation" T = 0
I didn't specify the context_window when I run : from llama_index import ServiceContext service_context = ServiceContext.from_defaults(chunk_size=1000, llm_predictor=llm, embed_model=embed_model) from llama_index import set_global_serivce_context set_global_serivce_context(service_context)
I get AttributeError: 'NoneType' object has no attribute 'context_window'
I tried to add context_window to falcon parameters but it looks like It doesn't have TypeError: RWForCausalLM.init() got an unexpected keyword argument 'context_window'