openchat/openchat_3.5 · Hugging Face

At a glance

The community members are discussing the possibility of using the openchat/openchat_3.5 model with the LlamaIndex library. One community member suggests using the HuggingFaceLLM abstraction, and another asks about using the TheBloke/openchat_3.5-GGUF model specifically. The community members provide some code examples, but encounter an error indicating that the specified model does not have the required files. Another community member mentions having success with using LlamaCPP, but there is no explicitly marked answer in the comments.

Useful resources

LLeMoussel

Is it possible to use openchat /openchat _3 .5 with LlamaIndex?
How to do?

5 comments

rravitheja

You can HuggingFaceLLM abstraction - https://docs.llamaindex.ai/en/stable/module_guides/models/llms/usage_custom.html#example-using-a-huggingface-llm

LLeMoussel

Thanks, and for TheBloke /openchat _3 .5-GGUF?

WWhiteFang_Jr

It will work. You just need to pass in the model name

LLeMoussel

Plain Text

llm = HuggingFaceLLM(
    model_name="TheBloke/openchat_3.5-GGUF",
    tokenizer_name="TheBloke/openchat_3.5-GGUF",
    query_wrapper_prompt=PromptTemplate("GPT4 User: {prompt}<|end_of_turn|>GPT4 Assistant:"),
    context_window=4096,
    max_new_tokens=1024,
    generate_kwargs={"temperature": 0.1, "top_k": 50, "top_p": 0.95, "do_sample": True},
    device_map="auto",
)

Got error:

TheBloke/openchat_3.5-GGUF does not appear to have a file named pytorch_model.bin, tf_model.h5, model.ckpt or flax_model.msgpack.

LLeMoussel

I had success with the use of LlamaCPP

Add a reply

Find answers from the community

openchat/openchat_3.5 · Hugging Face