Non OpenAI

May LLAMA index be used with other models than Openai? Let us say local LLAMA. How to do it? Thanks

15 comments

hey!
Yes you can use LlamaIndex with other LLMs very easily. And we have a lot of integrations with diff LLMs

Checkout this guide here for available Integrations: https://docs.llamaindex.ai/en/stable/module_guides/models/llms.html#modules

Also here is an example of Integrating Llama2 in LlamaIndex: https://docs.llamaindex.ai/en/stable/examples/llm/llama_2_llama_cpp.html

kkm

Hey! Thanks for reply. Is there an example of using any llama packs with LLAMA 2 locally?

WWhiteFang_Jr

Not sure if there is, But you can download the pack and replace the llm part with your newly created Llama2 llm object and it will work

kkm

Thanks, please tell: Let us say I wan to use LLAMA 2 with this pack, what should I replace
https://llamahub.ai/l/llama_packs-resume_screener?from=all

WWhiteFang_Jr

Here is the source code for this pack: https://github.com/run-llama/llama-hub/blob/e6960d4c64f053d32f1e92aa3a587f7b045cbfff/llama_hub/llama_packs/resume_screener/base.py#L58

You can pass your own llm here

WWhiteFang_Jr

Also pass embed model in there otherwise it will start looking for OpenAI key for embedding. Code would change to something like this:

Plain Text

class ResumeScreenerPack(BaseLlamaPack):
    def __init__(
        self, job_description: str, criteria: List[str], llm: Optional[LLM] = None  # PASS YOUR LLAMA2 LLM
    ) -> None:
        self.reader = PDFReader()
        llm = llm or OpenAI(model="gpt-4")    
        service_context = ServiceContext.from_defaults(llm=llm, embed_model='local') # ADD EMBED_MODEL to avoid falling on OpenAI 
        self.synthesizer = TreeSummarize(
            output_cls=ResumeScreenerDecision, service_context=service_context
        )
        criteria_str = _format_criteria_str(criteria)
        self.query = QUERY_TEMPLATE.format(
            job_description=job_description, criteria_str=criteria_str
        )

kkm

Thanks! Mby there is guide to integrating local ollama running on localhost with llama index?

kkm

@WhiteFang_Jr very strange errors appear when I try llam2 and mistral:instruct https://github.com/klimovskis/Resume_screening_using_Ollama_and_Llama_index/tree/main

kkm

May you please assist?

WWhiteFang_Jr

Sure, what error are you getting?

kkm

Thanks for reply! Here are the errors. They are different if launching llama2 and mistral:instruct. Here is for mistral:instruct

Attachments

WWhiteFang_Jr

I think open source model is not able to convert the answer into pydantic format.

Open source models are not good for these cases.

WWhiteFang_Jr

Sharing the compatibility report for open source llm.
https://docs.llamaindex.ai/en/stable/module_guides/models/llms.html#open-source-llms
You could see that llama2 and mistral are not very good at handling pyndantic scenarios.

kkm

Thanks, I see. So what sould be your approach?

WWhiteFang_Jr

If you check the compatibility report, you'll find that there are some open-source llms that are capable of handling pydantic output.

You can try those and see if any of them works for you

Add a reply

Find answers from the community

Non OpenAI