Find answers from the community

Updated 4 months ago

Huggingface inference

At a glance

The community member is wondering if there is a way to use LlamaIndex with the Hugging Face Inference API, such as with the falcon-7b-instruct model. Another community member responds that yes, this is possible by setting up the language model using the LangChain modules, and then using it in a ServiceContext from the LlamaIndex library, adjusting the context window and chunk size as needed.

ccodydh

Hello! I'm wondering if there's any way to use LlamaIndex with the Hugging Face Inference API, for example to use falcon-7b-instruct?

1 comment

LLogan M

Yes! You can setup the llm using the langchain modules, then just throw it into a service context

Plain Text

from llama_index import ServiceContext, set_global_service_context

llm = <setup from langchain>

# adjust context window and lower chunk size since the window is smaller
service_context = ServiceContext.from_defaults(llm=llm, context_window=2048, chunk_size=512)

set_global_service_context(service_context)

Add a reply