Find answers from the community

Updated 3 months ago

I'm trying to execute the sample code in

I'm trying to execute the sample code in https://docs.llamaindex.ai/en/stable/examples/multi_modal/gpt4v_multi_modal_retrieval.html and facing the following error. Can you think of anything that might have caused it?

​query_engine = index.as_query_engine(
multi_modal_llm=openai_mm_llm, text_qa_template=qa_tmpl
)

---------------------------------------------------------------------------
AssertionError Traceback (most recent call last)
Cell In[34], line 1
----> 1 query_engine = index.as_query_engine(
2 multi_modal_llm=openai_mm_llm, text_qa_template=qa_tmpl
3 )

File ~/.cache/pypoetry/virtualenvs/llama-index-gTaTDIvJ-py3.8/lib/python3.8/site-packages/llama_index/core/indices/multi_modal/base.py:155, in MultiModalVectorStoreIndex.as_query_engine(self, llm, kwargs) 152 retriever = cast(MultiModalVectorIndexRetriever, self.as_retriever(kwargs))
154 llm = llm or llm_from_settings_or_context(Settings, self._service_context)
--> 155 assert isinstance(llm, MultiModalLLM)
157 return SimpleMultiModalQueryEngine(
158 retriever,
159 multi_modal_llm=llm,
160 **kwargs,
161 )

AssertionError:
L
s
5 comments
Did you mix legacy imports with non-legacy imports?
According to the sample code, the legacy import is not used in this part as follows.
However, other part is used mixed legacy and non-legacy import here and there.
Do you think this mix is ​​affecting the error?

the part caused error
---------------------------------------------------
from llama_index.core import PromptTemplate
from llama_index.core.query_engine import SimpleMultiModalQueryEngine

qa_tmpl_str = (
"Context information is below.\n"
"---------------------\n"
"{context_str}\n"
"---------------------\n"
"Given the context information and not prior knowledge, "
"answer the query.\n"
"Query: {query_str}\n"
"Answer: "
)
qa_tmpl = PromptTemplate(qa_tmpl_str)

query_engine = index.as_query_engine(
multi_modal_llm=openai_mm_llm, text_qa_template=qa_tmpl
)

query_str = "Tell me more about the Porsche"
response = query_engine.query(query_str)
---------------------------------------------------


the part using mixed import
----------------------------------------------------
from llama_index.core.indices import MultiModalVectorStoreIndex
from llama_index.vector_stores.qdrant import QdrantVectorStore
from llama_index.core import SimpleDirectoryReader, StorageContext

import qdrant_client
from llama_index.core import SimpleDirectoryReader

Create a local Qdrant vector store

client = qdrant_client.QdrantClient(path="qdrant_mm_db")

text_store = QdrantVectorStore(
client=client, collection_name="text_collection"
)
image_store = QdrantVectorStore(
client=client, collection_name="image_collection"
)
storage_context = StorageContext.from_defaults(
vector_store=text_store, image_store=image_store
)

Create the MultiModal index

documents = SimpleDirectoryReader("./mixed_wiki/").load_data()
index = MultiModalVectorStoreIndex.from_documents(
documents,
storage_context=storage_context,
)
----------------------------------------------------
ah, theres no .legacy imports there 🤔
openai_mm_llm -- how did you set this up? I don't see it
Thank you for help me.

Thank you for help me.

I execute the following porcedure refering https://docs.llamaindex.ai/en/stable/getting_started/installation.html.

git clone https://github.com/jerryjliu/llama_index.git
pip install poetry
poetry shell
(llama-index-py3.8) bash-4.2$ poetry install
(llama-index-py3.8) bash-4.2$ poetry add --dev ipykernel
(llama-index-py3.8) bash-4.2$ ipython kernel install --user --name=llama-index

Also, I installed other packages as follows refering https://docs.llamaindex.ai/en/stable/examples/multi_modal/gpt4v_multi_modal_retrieval.html.

%pip install llama-index-multi-modal-llms-openai
%pip install llama-index-vector-stores-qdrant
%pip install llama_index ftfy regex tqdm
%pip install git+https://github.com/openai/CLIP.git
%pip install torch torchvision
%pip install matplotlib scikit-image
%pip install -U qdrant_client


Then, I made a notebook and select the kernel "llama-index" which I made above.
Other part of the sample code was successfully done.
But the most important part "Multi-Modal RAG Querying" was finished with the error....
Add a reply
Sign up and join the conversation on Discord