Question about

Plain Text

index.as_chat_engine(chat_mode="best")

Other than ReAct and OpenAI Chat Engine, is it possible to use Gemini?

30 comments

LLogan M

yea, gemini would use the react chat engine (or any other chat engine other than openai agent)

sswagz0521

@Logan M
Is there any documentation anywhere so I can force Gemini to be used as the Chat Engine? It seems like it's still using OpenAI in my case.

LLogan M

set it as the global default or pass it into the constructor

Plain Text

from llama_index.core import Settings

Settings.llm = ...

Plain Text

index.as_chat_engine(llm=llm, ...)

sswagz0521

Plain Text

if (model_choice == 'Gemini'):
    Settings.llm = Gemini(model="gemini-pro",
            temperature = temperature,
            system_prompt = system_prompt)

st.session_state.chat_engine = index.as_chat_engine(chat_mode = "best", llm = Settings.llm, context_prompt = (system_prompt), verbose=True)

sswagz0521

<llama_index.agent.openai.base.OpenAIAgent object at 0x7fdbae1cf760>

sswagz0521

It's still using OpenAlAgent.

LLogan M

sus lol

pip freeze | grep llama-index -- what does this show?

sswagz0521

Attachment

LLogan M

Are you sure its using openai for the LLM? It would be using it for embeddings still, but the LLM should be gemini

you can confirm this by print(chat_engine.worker._llm)

sswagz0521

Attachment

sswagz0521

Given it gives an error, I'm pretty sure it's still using OpenAIAgent.

LLogan M

oh spooky, why is it using openai agent

LLogan M

should be react agent

sswagz0521

Plain Text

st.session_state.chat_engine = index.as_chat_engine(chat_mode = "best", llm = Settings.llm, context_prompt = (system_prompt), verbose=True)

sswagz0521

In this code, I did have chat_mode as 'best'

sswagz0521

Does that default point to OpenAI?

LLogan M

it will try to init openai with the given llm (which will fail without openai) and then it will switch to react

LLogan M

lemme try to replicate... I don't really see how this is possible, at least by looking at the source code

LLogan M

I feel like something is maybe wonky in your streamlit/session state? This works fine locally for me

Plain Text

>>> from llama_index.core import VectorStoreIndex, Document, Settings
>>> from llama_index.llms.ollama import Ollama
>>> llm = Ollama(model="starling-lm", request_timeout=3000)
>>> Settings.llm = llm
>>> index = VectorStoreIndex.from_documents([Document.example()])
>>> chat_engine = index.as_chat_engine(chat_mode="best", llm=Settings.llm)
>>> chat_engine
<llama_index.core.agent.react.base.ReActAgent object at 0x7f6b5abb7fd0>
>>> chat_engine.agent_worker._llm
Ollama(callback_manager=<llama_index.core.callbacks.base.CallbackManager object at 0x7f6c169baf80>, system_prompt=None, messages_to_prompt=<function messages_to_prompt at 0x7f6c16a2ab00>, completion_to_prompt=<function default_completion_to_prompt at 0x7f6c168c9d80>, output_parser=None, pydantic_program_mode=<PydanticProgramMode.DEFAULT: 'default'>, query_wrapper_prompt=None, base_url='http://localhost:11434', model='starling-lm', temperature=0.75, context_window=3900, request_timeout=3000.0, prompt_key='prompt', additional_kwargs={})
>>>

LLogan M

(I dont have access to gemini, so picked something else)

sswagz0521

Thank you for testing it, let me take a look.

sswagz0521

So this is the output of the Settings.llm after I set it to Gemini

Attachment

LLogan M

A little truncated, but looks correct (I see gemini-pro at the end, which is a good sign)

sswagz0521

Attachment

sswagz0521

Maybe the once chat_engine is set up in the same session it doesn't change?

sswagz0521

And I assume the default is OpenAI.

sswagz0521

Aha, I found the bug.

sswagz0521

Exactly what I thought.

sswagz0521

I have this code that lets users to pick a LLM Model, but default as GPT.

When it first initialize the Streamlit session, it defaults to GPT and doesn't change when you change the model setting.

LLogan M

ahh interesting 🤔

Add a reply

Find answers from the community

Question about