Service context

At a glance

The community member is trying to use the Mistral language model in the LlamaIndex library, but is encountering an error that appears to be related to the default use of the OpenAI model. The community members discuss whether Mistral is supported in the service context, with one member suggesting that it may not be supported based on the API reference, and another stating that it is supported, but the API reference is incorrect. The community members try various approaches, such as using the HuggingFace API, but are unable to get it working. There is no explicitly marked answer in the comments.

Useful resources

ssysfor

Curious if it's possible to use Mistral for service context? If I replace with OpenAI it works just fine.

Plain Text

llm = MistralAI(model="mistral-tiny", 
                api_key=MISTRAL_API_KEY)
....
print(type(llm))
<class 'llama_index.llms.mistral.MistralAI'>
....

service_context = ServiceContext.from_defaults(llm=llm,
                                    embed_model=embed_model,
                                    system_prompt=SPF)
....
index = VectorStoreIndex.from_documents(
    documents=[web_docs, pdf_docs],
    service_context=service_context,
    storage_context=storage_context,
    show_progress=True
)

It's throwing an OpenAI error. Appears to be trying to default to OpenAI: Could not load OpenAI model. If you intended to use OpenAI, please check your OPENAI_API_KEY.

5 comments

LLogan M

Are you passing the service context to the other modules in your code?

(Some modules also take the LLM directly as a kwarg)

ssysfor

After doing some more reading I think it may not be supported. When looking at the service context classes for Service Context here are the LLMs listed:

https://docs.llamaindex.ai/en/stable/api_reference/llms.html

ssysfor

I guess it might be possible to call it via the huggingface api via HuggingFaceLLM - im going to try that. Id prefer to call mistral directly

LLogan M

No it's supported, api ref is just bad

ssysfor

Hmm. I couldn't get it working. Ill keep trying

Add a reply

Find answers from the community

Service context