Hello

TThomas1234

Hello,
While trying to use the new HuggingFaceLLM function with the Vector Store Index I get the following error:
'HuggingFaceLLM' object has no attribute 'predict'
Does anyone have an answer to this ?
I'm following this tutorial: https://gpt-index.readthedocs.io/en/v0.7.0/examples/customization/llms/SimpleIndexDemo-Huggingface_camel.html
Also I've added embedding in the service context using LangChainEmbedding since I was getting an Open AI API key error
Thanks

11 comments

LLogan M

hmm, can I see your code?

TThomas1234

Sure @Logan M

import logging
import sys

logging.basicConfig(stream=sys.stdout, level=logging.INFO)
logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))

from llama_index import VectorStoreIndex, SimpleDirectoryReader, ServiceContext, LangchainEmbedding
from llama_index.llms import HuggingFaceLLM
from llama_index.prompts.prompts import SimpleInputPrompt

load documents

documents = SimpleDirectoryReader('/data').load_data()

query_wrapper_prompt = SimpleInputPrompt(
"Below is an instruction that describes a task. "
"Write a response that appropriately completes the request.\n\n"
"### Instruction:\n{query_str}\n\n### Response:"
)
import torch
llm = HuggingFaceLLM(
# context_window=2048, Returns an error message
# max_new_tokens=256,
temperature=0.25,
do_sample=False,
query_wrapper_prompt=query_wrapper_prompt,
tokenizer_name="Writer/camel-5b-hf",
model_name="Writer/camel-5b-hf",
device_map="auto",
tokenizer_kwargs={"max_length": 2048},
model_kwargs={"torch_dtype": torch.float16}
)

from langchain.embeddings import HuggingFaceEmbeddings
embed_model = LangchainEmbedding(HuggingFaceEmbeddings(model_name = "sentence-transformers/all-mpnet-base-v2")) #I get an Open AI API error without

service_context = ServiceContext.from_defaults(chunk_size=512, llm=llm, embed_model = embed_model)

index = VectorStoreIndex.from_documents(documents, service_context=service_context)

query_engine = index.as_query_engine()
response = query_engine.query("What did the author do growing up?")

LLogan M

hmmm, that looks correct. I'll try to replicate in colab, one sec 🙂

LLogan M

while I'm waiting for models to download (lol), what version of llama-index do you have? pip show llama-index

TThomas1234

Version 0.7.2

LLogan M

that error makes no sense to me, at least from a code perspective.

I'm not able to replicate either 🙃 Can you try either updating to the latest version, or just for a sanity check, wrap the llm with the predictor class

Plain Text

from llama_index import LLMPredictor

llm_predictor = LLMPredictor(llm=llm)
service_context = ServiceContext.from_defaults(chunk_size=512, llm_predictor=llm_predictor, embed_model=embed_model)

TThomas1234

I'll try with the latest version of llama-index and i'll let you know, also do you know if this works with all models available on HuggingFace ?

TThomas1234

@Logan M Works fine in the version 0.7.5 !!

LLogan M

It does! Assuming the model from huggingface is a text-generation model 🙂 Some models will require different prompt settings and context window settings, which can hopefully be found in the model card or config.json files

LLogan M

Amazing!

TThomas1234

Thanks for your help

Add a reply

Find answers from the community

Hello

load documents