Find answers from the community

Updated 2 years ago

Higgingface LLM

@Logan M I'm trying to use the HuggingFaceHub from LangChain along with LLamaIndex. The solution that you shared on GitHub, I tried it but somehow it's not working for me. Can you please help me out?

Here's your github response:
https://github.com/jerryjliu/llama_index/issues/3290#issuecomment-1546914037

11 comments

LLogan M

Ah right right. What's the error you are getting with this setup?

ddatum

Umm it's not throwing any error but just outputting "Empty Response"

LLogan M

Hmm. I'm not 100% sure how huggingface hub works. What are your prompt helper settings?

LLogan M

Normally on local hughingface models you'd set max_tokens or max_new_tokens, not sure what it defaults to doing 😅

ddatum

Well I can send you once I'm back in front of my pc 😅

ddatum

So here's my prompt helper:

Plain Text

prompt_helper = PromptHelper(
     context_window=context_window,
      num_output=num_output,
      chunk_size_limit=chunk_size_limit
   )

LLogan M

But what are the values being used there? 👀

ddatum

Plain Text

context window=4096
num_output=256
chunk_size_limit=2048

LLogan M

Ooo there's one issue. I think falcon only supports a context window of 2048.

Try setting the context window to 2048, and the chunk size to 512 maybe?

LLogan M

Also put chunk_size=512 directly in the service context, to also set the node parser to use that chunk size as well

ddatum

Okay, will do that then 😀

Add a reply