ValueError: Unrecognized configuration

FLAN-T5 is an encoder-decoder model, rather than a decoder model

A ton of logic in llama-index is built around the idea of decoder models. T5 is the only LLM architecture out there not using a pure decorder format.

You'll have to implement a customLLM class from scratch if you want to use T5
https://docs.llamaindex.ai/en/stable/module_guides/models/llms/usage_custom.html#example-using-a-custom-llm-model-advanced

In addition, since T5 predicts all output tokens in one step (rather than one at at time), configure max_new_tokens on the LLM Metadata to be zero, otherwise llama-index will "leave room" in the input for the LLM to predict tokens (which is only needed for decoder models)

How about Mistral models

I tried running them but failed there too

mistral should be fine. We have a demo here of the setup

Thank you

Let me try this

Will it work for the latest mixtral model too?

It should, they have the same prompt format from my understanding

it does there are some display issue and one cell does not work, unfortunately i am hitting same issue with
raw_nodes_2021 = node_parser.get_nodes_from_documents(documents, service_context=service_context)
pickle.dump(raw_nodes_2021, open("2021_nodes.pkl", "wb"))

ValueError: No API key found for OpenAI.
Please set either the OPENAI_API_KEY environment variable or openai.api_key prior to initialization.
API keys can be found or created at https://platform.openai.com/account/api-keys


During handling of the above exception, another exception occurred:

ValueError Traceback (most recent call last)
/usr/local/lib/python3.10/dist-packages/llama_index/embeddings/utils.py in resolve_embed_model(embed_model)
48 validate_openai_api_key(embed_model.api_key)
49 except ValueError as e:
---> 50 raise ValueError(
51 "\n**\n"
52 "Could not load OpenAI embedding model. "

ValueError:
**
Could not load OpenAI embedding model. If you intended to use OpenAI, please check your OPENAI_API_KEY.
Original error:
No API key found for OpenAI.
Please set either the OPENAI_API_KEY environment variable or openai.api_key prior to initialization.
API keys can be found or created at https://platform.openai.com/account/api-keys

Consider using embed_model='local'.
Visit our documentation for more embedding options: https://docs.llamaindex.ai/en/stable/module_guides/models/embeddings.html#modules
**

otherwise the local models (there is also an embedding model) work on T4

the node parser is
from llama_index.node_parser import (
UnstructuredElementNodeParser,
)

node_parser = UnstructuredElementNodeParser()

pass in the LLM to the node parser (it's using the LLM to generate summaries of tables)

what am I missing?

llm = HuggingFaceLLM(
model_name="mistralai/Mistral-7B-Instruct-v0.1",
tokenizer_name="mistralai/Mistral-7B-Instruct-v0.1",
query_wrapper_prompt=PromptTemplate("<s>[INST] {query_str} [/INST] </s>\n"),
context_window=3900,
max_new_tokens=256,
model_kwargs={"quantization_config": quantization_config},
# tokenizer_kwargs={},
generate_kwargs={"temperature": 0.2, "top_k": 5, "top_p": 0.95},
device_map="auto",
)

raw_nodes_2021 = node_parser.get_nodes_from_documents(documents,llm=llm,embed_model="local:BAAI/bge-small-en-v1.5")

throws that error

also i did

service_context = ServiceContext.from_defaults(llm=llm, embed_model="local:BAAI/bge-small-en-v1.5")

---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
Input In [7], in <cell line: 14>()
4 from llama_index.llms import HuggingFaceLLM
6 quantization_config = BitsAndBytesConfig(
7 load_in_4bit=True,
8 bnb_4bit_compute_dtype=torch.float16,
9 bnb_4bit_quant_type="nf4",
10 bnb_4bit_use_double_quant=True,
11 )
---> 14 llm = HuggingFaceLLM(
15 model_name="mistralai/Mistral-7B-Instruct-v0.1",
16 tokenizer_name="mistralai/Mistral-7B-Instruct-v0.1",
17 query_wrapper_prompt=PromptTemplate("<s>[INST] {query_str} [/INST] </s>\n"),
18 context_window=3900,
19 max_new_tokens=256,
20 model_kwargs={"quantization_config": quantization_config},
21 # tokenizer_kwargs={},
22 generate_kwargs={"temperature": 0, "top_k": 5, "top_p": 0.95},
23 device_map="auto",
24 )

File ~/anaconda3/lib/python3.9/site-packages/llama_index/llms/huggingface.py:175, in HuggingFaceLLM.init(self, context_window, max_new_tokens, query_wrapper_prompt, tokenizer_name, model_name, model, tokenizer, device_map, stopping_ids, tokenizer_kwargs, tokenizer_outputs_to_remove, model_kwargs, generate_kwargs, is_chat_model, callback_manager, system_prompt, messages_to_prompt, completion_to_prompt, pydantic_program_mode, output_parser)
169 raise ImportError(
170 f"{type(self).name} requires torch and transformers packages.\n"
171 "Please install both with pip install transformers[torch]."
172 ) from exc
174 model_kwargs = model_kwargs or {}
--> 175 self._model = model or AutoModelForCausalLM.from_pretrained(
176 model_name, device_map=device_map, **model_kwargs
177 )
179 # check context_window
180 config_dict = self._model.config.to_dict()

File ~/anaconda3/lib/python3.9/site-packages/transform
KeyError: 'mistral'

Update your transformers install

Tried running this colab notebook but gives error

Which version

No idea. Should be latest

pip install --upgrade transformers

Thanks

ValueError: The current device_map had weights offloaded to the disk. Please provide an offload_folder for them. Alternatively, make sure you have safetensors installed if the model you are using offers the weights in this format.

https://docs.llamaindex.ai/en/stable/examples/customization/llms/SimpleIndexDemo-Huggingface_camel.html when ran this

That means you didn't have enough memory to fit the weights

Ok

Can using offloader help here

it will be too slow, I wouldn't even try to enable it

Ok

Thanks for all your help

Which ec2 instance I shall use

How much RAM will be good