Find answers from the community

Updated 3 months ago

Question about

Question about
Plain Text
index.as_chat_engine(chat_mode="best")

Other than ReAct and OpenAI Chat Engine, is it possible to use Gemini?
L
s
30 comments
yea, gemini would use the react chat engine (or any other chat engine other than openai agent)
@Logan M
Is there any documentation anywhere so I can force Gemini to be used as the Chat Engine? It seems like it's still using OpenAI in my case.
set it as the global default or pass it into the constructor

Plain Text
from llama_index.core import Settings

Settings.llm = ...


Or

Plain Text
index.as_chat_engine(llm=llm, ...)
Plain Text
if (model_choice == 'Gemini'):
    Settings.llm = Gemini(model="gemini-pro",
            temperature = temperature,
            system_prompt = system_prompt)

st.session_state.chat_engine = index.as_chat_engine(chat_mode = "best", llm = Settings.llm, context_prompt = (system_prompt), verbose=True)
<llama_index.agent.openai.base.OpenAIAgent object at 0x7fdbae1cf760>
It's still using OpenAlAgent.
sus lol

pip freeze | grep llama-index -- what does this show?
Are you sure its using openai for the LLM? It would be using it for embeddings still, but the LLM should be gemini

you can confirm this by print(chat_engine.worker._llm)
Given it gives an error, I'm pretty sure it's still using OpenAIAgent.
oh spooky, why is it using openai agent
should be react agent
Plain Text
st.session_state.chat_engine = index.as_chat_engine(chat_mode = "best", llm = Settings.llm, context_prompt = (system_prompt), verbose=True)
In this code, I did have chat_mode as 'best'
Does that default point to OpenAI?
it will try to init openai with the given llm (which will fail without openai) and then it will switch to react
lemme try to replicate... I don't really see how this is possible, at least by looking at the source code
I feel like something is maybe wonky in your streamlit/session state? This works fine locally for me

Plain Text
>>> from llama_index.core import VectorStoreIndex, Document, Settings
>>> from llama_index.llms.ollama import Ollama
>>> llm = Ollama(model="starling-lm", request_timeout=3000)
>>> Settings.llm = llm
>>> index = VectorStoreIndex.from_documents([Document.example()])
>>> chat_engine = index.as_chat_engine(chat_mode="best", llm=Settings.llm)
>>> chat_engine
<llama_index.core.agent.react.base.ReActAgent object at 0x7f6b5abb7fd0>
>>> chat_engine.agent_worker._llm
Ollama(callback_manager=<llama_index.core.callbacks.base.CallbackManager object at 0x7f6c169baf80>, system_prompt=None, messages_to_prompt=<function messages_to_prompt at 0x7f6c16a2ab00>, completion_to_prompt=<function default_completion_to_prompt at 0x7f6c168c9d80>, output_parser=None, pydantic_program_mode=<PydanticProgramMode.DEFAULT: 'default'>, query_wrapper_prompt=None, base_url='http://localhost:11434', model='starling-lm', temperature=0.75, context_window=3900, request_timeout=3000.0, prompt_key='prompt', additional_kwargs={})
>>> 
(I dont have access to gemini, so picked something else)
Thank you for testing it, let me take a look.
So this is the output of the Settings.llm after I set it to Gemini
Attachment
image.png
A little truncated, but looks correct (I see gemini-pro at the end, which is a good sign)
Maybe the once chat_engine is set up in the same session it doesn't change?
And I assume the default is OpenAI.
Aha, I found the bug.
Exactly what I thought.
I have this code that lets users to pick a LLM Model, but default as GPT.

When it first initialize the Streamlit session, it defaults to GPT and doesn't change when you change the model setting.
ahh interesting πŸ€”
Add a reply
Sign up and join the conversation on Discord