LlamaIndex

Log inLog into community

Find answers from the community

s

F

Y

a

P

Updated 10 months ago

I'm struggling to understand how

I'm struggling to understand how

·

I'm struggling to understand how LangChainLLM() is fundamentally different than other llms like LlamaCPP() or SagemakerLLM(), specifically in that you cannot set messages_to_prompt or completion_to_prompt on LangChainLLM but you can on the others.

After looking through the source code it looks like LangChainLLM is the only one that extends LLM instead of CustomLLM.

My use case is the following:

I'm attempting to use an arbitrary TGI Hugging Face model to do streaming chat as shown below

Plain Text

hf = HuggingFaceTextGenInference(
                    inference_server_url="https://api-inference.huggingface.co/models/HuggingFaceH4/zephyr-7b-beta",
                    max_new_tokens=512,
                    top_k=10,
                    top_p=0.95,
                    typical_p=0.95,
                    temperature=0.01,
                    repetition_penalty=1.03,
                )
                prompt_style = get_prompt_style(settings.huggingface.prompt_style)
                self.llm = LangChainLLM(llm=hf)

Because I'm using an arbitrary model, I need to be able to change the messages_to_prompt function. Is there a way to do this?

L

a

51 comments

LangChainLLM is just a wrapper around llms from langchain, so that they have the expected interface our our LLM class

Looks like an oversight with accepting those kwargs

actually

It looks fine to me

Attachment

You should be able to do LangChainLLM(hf, messages_to_prompt=my_function, ...)

Attachment

Screenshot_2024-01-03_at_7.38.29_PM.png

Am i looking at an old version?

Yea you are

Maybe update your installation? It was added sometime around... 0.9.10 I want to say?

Can't remember

i'm working on privateGPT and they are on 9.3,

llama-index = { extras = ["local_models"], version = "0.9.3" }

will it work to pass it through as kwargs or do I need to campaign them to update?

Technically you can just pip install llama-index==0.9.25 and it should install fine (pip will complain though with a warning)

They probably should bump the dependency though. There shouldn't be any breaking changes

I really appreciate your help on this. After updating this morning I looked through the source to see if the modifications I had imagined were actually what got implemented.

Though you can now pass in messages_to_prompt to LangChainLLM with no issues, it doesn't appear that its used anywhere.

The LangChainLLM.complete is straightforward as it calls _llm.predict() which eventually calls the passed in completion_to_prompt function.

LangChainLLM.chat is more confusing though as it calls predict_messages() which seems to be defined on BaseLLM and never in the chain seems to call messages_to_prompt.

I may be misunderstanding one of the abstraction layers though, would you be willing to show me how messages_to_prompt ever gets called? In practice it doesn't appear to be getting called ether.

Thanks so much

ha yes good point, its not used 🤦‍♂️

will update that

hmm little tricky though tbh. I can make it use completion_to_prompt() but not sure how messages_to_prompt will be ever used.

chat() will only be called for chat models that support messages as input in the first place 🤔

so no way to grab and format the messages for non-chat models. At least in the LLM class

actually, maybe this will still work, just needs a bit more tweaking

What I'm looking for is a way to pass in a list of messages to the LLM via stream_chat(messages) and have that function do something like the following.

Plain Text

def stream_chat(
        self, messages: Sequence[ChatMessage], **kwargs: Any
    ) -> ChatResponseGen:
        prompt = self.messages_to_prompt(messages)
        completion_response = self.stream_complete(prompt, formatted=True, **kwargs)
        return stream_completion_response_to_chat_response(completion_response)

Attachment

Screenshot_2024-01-04_at_9.49.41_AM.png

yea the code needs to be updated so that if chat() is called, but its not a chat model, that it gets routed to complete()

doing that now

Ah yes yes

Really thought I was crazy for a hot minute, thank you for working on this

while I'm at, lets see if langchain finally added async to their base classes

ayyy they did, nice

ugh ok thats a lot more work though lol one thing at a time

Do you mind trying it out?

pip install git+https://github.com/run-llama/llama_index.git@logan/use_messages_to_prompt_langchain

https://github.com/run-llama/llama_index/pull/9844

On it, I'll let you know

Very curious to see your solution, it wasn't obvious to me through its probably because I don't really understand the BasePromptTemplate class's function

it was a pretty simple change thankfully 🙂

is_formatted does the trick 🫡

nice perfect 🙂

I haven't run it yet! Just had a look at the code cause i was curious.

Almost done setting up a google collab to test

oh lol I pulled the trigger and merged it lol

pip install git+https://github.com/run-llama/llama_index.git to try it now

Haha oops 🤦‍♂️

sorry about that lol

all good, I have like 95% confidence it will work hahaha

I'm kinda new to python and privateGPT uses poetry which I have no experience with so I wasn't sure how to do a drop-in replacement

hopefully this collab thing works

generally pip and poetry are compatible.

I might use poetry shell and poetry install to setup an initial env, and then use pip install for on-the-fly changes

🫡 🫡 🫡

Attachment

Screenshot_2024-01-04_at_10.27.00_AM.png

You're the man, thank you

Did LLM change to BaseLLM from 9.3 to current?

This throws an error
from llama_index.llms.base import LLM

but i see that file exports BaseLLM, i assume those are equivilant?

They are basically the same

BaseLLM is the raw interface, same as the old LLM

the new LLM adds some extra sugar on top

They had to be seperate files to avoid circular imports

Basically LLMPredictor was deprecated, and I moved its interface into the LLM class itself

ah i see, LLM is just not defined in the base.py file anymore, now its got its own llm.py file

yup 👍

Add a reply

Sign up and join the conversation on Discord