Has anyone run with mistral? Getting

At a glance

Has anyone run with mistral? Getting some validation errors when I input my own model name in the openai constructor.

Plain Text

model='Mistral-7B-Instruct-v0.1'
llm = OpenAI(api_base=api_base, api_key=api_key, model=model)

returns

Plain Text

ValueError: Unknown model 'Mistral-7B-Instruct-v0.1'. Please provide a valid OpenAI model name in: gpt-4, gpt-4-32k, gpt-4-0613, gpt-4-32k-0613, gpt-4-0314, gpt-4-32k-0314, gpt-3.5-turbo, gpt-3.5-turbo-16k, gpt-3.5-turbo-0613, gpt-3.5-turbo-16k-0613, gpt-3.5-turbo-0301, text-davinci-003, text-davinci-002, gpt-3.5-turbo-instruct, text-ada-001, text-babbage-001, text-curie-001, ada, babbage, curie, davinci, gpt-35-turbo-16k, gpt-35-turbo

21 comments

DDangFutures

you are using the openai class thats why trying using huggingface or vllm https://gpt-index.readthedocs.io/en/stable/examples/vector_stores/SimpleIndexDemoLlama-Local.html change the model name and config to what you need

here is the vllm using langchain https://python.langchain.com/docs/integrations/llms/vllm

SSteven Tobias

Hmm.... It's not local though

SSteven Tobias

this is on a remote server

SSteven Tobias

mistral docs say I should be able to just drop in mistral and a different baseurl

SSteven Tobias

I can do a simple predcit, but things fall apart when I try to use things like text to sql

SSteven Tobias

I think it's b/c the tokenizer mappings etc aren't set up

bbmax

You can't use the OpenAI class from LlamaIndex

SSteven Tobias

@bmax is there another suggestion?

SSteven Tobias

pretty new to this library, posting in help channel... b/c I need some help lol

SSteven Tobias

even if the suggestion is to just extend the CustomLLM class that's fine. just dont wanna spin my gears if it's already solved

bbmax

I actually don't know! I thought @DangFutures had some good suggestions. https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1 looks like it's on huggingface, so if you don't use huggingface, and instead have it hosted remotely?

If mistral says you can drop in a different base url and it uses same API then you can change OPENAI BASE_URL credentials here:

https://github.com/run-llama/llama_index/blob/main/llama_index/llms/openai.py#L92

bbmax

@Steven Tobias hope that helps!

bbmax

oh you are changing the base_url in your post above

bbmax

hmm yeah, it looks like that error is thrown in
openai_modelname_to_contextsize function

SSteven Tobias

yeah it seems like it's something to do with all the extra "magic" that comes along with some of the integrations here.

next time I get a chance to look at it, maybe the answer is just to fork into a new llm (on the llamaindex side). the API compat from mistral is great, but I think there's some missing pieces for more complete use cases like llama index

SSteven Tobias

ty for th ehelp

bbmax

should be pretty easy!

DDangFutures

@Steven Tobias have you tried using the openai replacement from vllm

rravitheja

@Steven Tobias You cannot send mistral model name in openai class. You can use anyscale or replicate endpoints where mistral is hosted and use llamaindex connectors accordingly.

https://x.com/ravithejads/status/1714859246859235642?s=20 - here is one such example.

SSteven Tobias

tinkered a bit, got something working with vllm from langchain integration

bbmax

❤️

Add a reply

Find answers from the community

Has anyone run with mistral? Getting