Find answers from the community

Updated 2 months ago

Using Ollama with Llama_index

At a glance

The community members have questions about using Ollama with llama_index, such as how to switch between language models (like llama3.2 and phi), how to manage multiple conversations, and how to handle vector stores and indexes. The comments discuss topics like settings, chat engines, and the need to potentially create multiple vector stores and indexes for different language models. While there is no explicitly marked answer, the community members seem to be working through these issues and trying to understand the best practices for managing multiple language models and conversations.

Useful resources
Hi, I have a few questions about using Ollama with llama_index.

If I am currently chatting with llama3.2 using:
llm = Ollama(model="llama3.2:latest")
and I want to switch to phi, should I do:
llm = Ollama(model="phi")?

If I want to continue the conversation with the previous llama3.2 instance after switching to phi, should I create two separate instances—one for llama3.2 and one for phi?

If I want to start a completely new chat with llama3.2, is it necessary to create a new instance for it?

If I have 5 different conversations (possibly using the same or different models), should I create 5 separate instances to manage them?

Thanks in advance for your help!
M
L
11 comments
How are they working with Settings, vector store, index, ....

If I switch to a new llm instance, Settings.llm is also required to change?
chat_engine = index.as_chat_engine(llm=xxxx, memory=xxx, ...)
Settings is just a global default. You can pass in to override locally, as your code sample above does
I have a question about vectorstore, as multiple models available in my program.

vectorstore = DuckDBVectorsStore.from_local(db_path)
index = VectorStoreIndex.from_vector_store(vectorstore)

The problem is that when I add documents to the DB, it also changes the data in the drive.

Supposing we use phi + DB + doc_A, then switch to llama + DB + doc_B. In my current program, the switching requires remove doc_A from DB and append doc_B to DB.

Is it possible to load and create multiple vectorstore, append/remove doc doesn't affect the DB in the drive?
I don't think I really understand 🤔
https://docs.llamaindex.ai/en/stable/module_guides/indexing/vector_store_index/
it is related to this one.

DuckDBVectorsStore.from_local(db_path)
db_path is the file path for pre-saved database.
vectorstore = DuckDBVectorsStore.from_local(db_path)
index = VectorStoreIndex.from_vector_store(vectorstore)
llama_chat_engine = index.as_chat_engine(llm=llama_lm, ...)
phi_chat_engine = index.as_chat_engine(llm=phi_lm, ...)
--------------------------------------------------------------------------------------
My code is here, I found change the argument llm doesn't change the chat_engine...

def init_llm(self):
names = [
"llama3.2",
"gemma2",
"mistral",
]

for name in names:
self.lm_dict[name] = Ollama(model=name, request_timeout=120.0)
self.cur_lm = self.lm_dict[names[0]]
self.cur_lm_name = names[0]


def set_current_lm(self, llm_name):

self.cur_lm_name = llm_name
self.cur_lm = self.lm_dict[llm_name]


self.memory = ChatMemoryBuffer.from_defaults(
token_limit=3000,
chat_store=self.chat_store,
chat_store_key=llm_name,
)

self.chat_engine = self.index.as_chat_engine(
chat_mode="context",
llm=self.cur_lm,
memory=self.memory,
system_prompt=(
"You are a helpful assistant which helps users to understand scientific knowledge"
"about biomechanics of injuries to human bodies."
),
verbose=True,
)
Should I create mult index for each language model....?
OK. I figure out. my previous question to the model after switching model is bad, so it generate unclear response...
Add a reply
Sign up and join the conversation on Discord