I am facing the following error when using cuda as a device map:
---------------------------------------------------------------------------
ValidationError Traceback (most recent call last)
Cell In[1], line 25
23 qa_template = Prompt(template)
24 # Initialize HuggingFaceLLM
---> 25 llm = HuggingFaceLLM(
26 context_window=1028,
27 max_new_tokens=256,
28 generate_kwargs={"temperature": 0.7, "do_sample": False},
29 system_prompt=system_prompt,
30 query_wrapper_prompt=query_wrapper_prompt,
31 tokenizer_name="StabilityAI/stablelm-tuned-alpha-3b",
32 model_name="StabilityAI/stablelm-tuned-alpha-3b",
33 device_map="cuda",
34 stopping_ids=[50278, 50279, 50277, 1, 0],
35 tokenizer_kwargs={"max_length": 1028},
36 # uncomment this if using CUDA to reduce memory usage
37 model_kwargs={"torch_dtype": torch.float16}
38 )


File /opt/conda/envs/sagemaker-distribution/lib/python3.8/site-packages/pydantic/main.py:341, in pydantic.main.BaseModel.init()

ValidationError: 1 validation error for HuggingFaceLLM
query_wrapper_prompt
str type expected (type=type_error.str)

i am trying to put {context_str} in a prompt for stablelm, but it is giving message that it is not initailzed. does this lm doesnt accept context string? i want it to answer specifically from the document and not from its prior know;edge

i want a fast free model from qa chatbot from huggingfacellm that gives a response in seconds

from llama_index import GPTVectorStoreIndex, SimpleDirectoryReader, SummaryPrompt
from llama_index import Prompt, ListIndex, SimpleDirectoryReader
Define your custom prompt
SUMMARY_PROMPT_TMPL = (
"We have provided context information below. \n"
"---------------------\n"
"{context_str}"
"\n---------------------\n"
"Given this information, please answer the question: {query_str}\n"
)
Load your documents
documents = SimpleDirectoryReader(input_files=["/content/Example.pdf"]).load_data()

SUMMARY_PROMPT = SummaryPrompt(SUMMARY_PROMPT_TMPL)

Use the custom prompt with TreeIndex
index_with_query = GPTVectorStoreIndex(documents, summary_template=SUMMARY_PROMPT)

Configure the query engine with the custom prompt
query_engine = index_with_query.as_query_engine(text_qa_template=SUMMARY_PROMPT)
response = query_engine.query("based on the given context what is the difference btween Bird and technical documentation?")
print(response)

the reponse is taking 20-30 mins to be generated eventhough the document is only 3 pages. what to do to make the response faster

@kapa.ai
how to use llama2 model with llama-index

Hello @Logan M , so I am noticing something weird happening. when I am using conda environment, and using huggingfacellm, an error pops out query_wrapper_prompt
str type expected (type=type_error.str). Whereas the same code is being used without conda env works properly, why is this issue happening?

@kapa.ai

query_wrapper_prompt = SimpleInputPrompt("<|USER|>{query_str}<|ASSISTANT|>")

Initialize HuggingFaceLLM

llm = HuggingFaceLLM(
context_window=1028,
max_new_tokens=256,
generate_kwargs={"temperature": 0.7, "do_sample": False},
system_prompt=system_prompt,
query_wrapper_prompt=query_wrapper_prompt,
tokenizer_name="StabilityAI/stablelm-tuned-alpha-3b",
model_name="StabilityAI/stablelm-tuned-alpha-3b",
device_map="cpu",
stopping_ids=[50278, 50279, 50277, 1, 0],
tokenizer_kwargs={"max_length": 1028},
# uncomment this if using CUDA to reduce memory usage
# model_kwargs={"torch_dtype": torch.float16}
)

ValidationError Traceback (most recent call last)
/tmp/ipykernel_910/4083314887.py in <cell line: 25>()
23 qa_template = Prompt(template)
24 # Initialize HuggingFaceLLM
---> 25 llm = HuggingFaceLLM(
26 context_window=1028,
27 max_new_tokens=256,

~/.conda/envs/default/lib/python3.9/site-packages/llama_index/llms/huggingface.py in init(self, context_window, max_new_tokens, system_prompt, query_wrapper_prompt, tokenizer_name, model_name, model, tokenizer, device_map, stopping_ids, tokenizer_kwargs, tokenizer_outputs_to_remove, model_kwargs, generate_kwargs, callback_manager)
148 self._stopping_criteria = StoppingCriteriaList([StopOnTokens()])
149
--> 150 super().init(
151 context_window=context_window,
152 max_new_tokens=max_new_tokens,

~/.conda/envs/default/lib/python3.9/site-packages/pydantic/main.cpython-39-x86_64-linux-gnu.so in pydantic.main.BaseModel.init()

ValidationError: 1 validation error for HuggingFaceLLM
query_wrapper_prompt
str type expected (type=type_error.str)

@kapa.ai this is my query wrapper: query_wrapper_prompt = SimpleInputPrompt("<|USER|>{query_str}<|ASSISTANT|>")

but iam getting this error:
-----------------------
ValidationError Traceback (most recent call last)
Cell In[1], line 25
23 qa_template = Prompt(template)
24 # Initialize HuggingFaceLLM
---> 25 llm = HuggingFaceLLM(
26 context_window=1028,
27 max_new_tokens=256,
28 generate_kwargs={"temperature": 0.7, "do_sample": False},
29 system_prompt=system_prompt,
30 query_wrapper_prompt=query_wrapper_prompt,
31 tokenizer_name="StabilityAI/stablelm-tuned-alpha-3b",
32 model_name="StabilityAI/stablelm-tuned-alpha-3b",
33 device_map="cuda",
34 stopping_ids=[50278, 50279, 50277, 1, 0],
35 tokenizer_kwargs={"max_length": 1028},
36 # uncomment this if using CUDA to reduce memory usage
37 model_kwargs={"torch_dtype": torch.float16}
38 )


File /opt/conda/envs/sagemaker-distribution/lib/python3.8/site-packages/pydantic/main.py:341, in pydantic.main.BaseModel.init()

ValidationError: 1 validation error for HuggingFaceLLM
query_wrapper_prompt
str type expected (type=type_error.str)