Is there a good example / cookbook for

kkev

Is there a good example / cookbook for multi-vector / recursive retriever + multi-modal RAG using llama-index?

Here's an LangChain example: https://github.com/langchain-ai/langchain/blob/master/cookbook/Multi_modal_RAG.ipynb

21 comments

LLogan M

Have you seen this page? https://docs.llamaindex.ai/en/stable/use_cases/multimodal/

This is pretty comprehensive for example
https://docs.llamaindex.ai/en/stable/examples/multi_modal/ollama_cookbook/?h=multimodal

kkev

@Logan M I was hoping to load the llava model from HF instead of ollama . any idea how it would work in this example? https://docs.llamaindex.ai/en/stable/examples/multi_modal/ollama_cookbook/?h=multimodal

kkev

doesn't look like I can simply pass the model from HF to multi_modal_llm parameter.

kkev

Plain Text

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-58-6d30c419b9a0> in <cell line: 1>()
----> 1 response = mm_program(query_str="What was the value of Non Revenue Units in Apr 2022?")
      2 print(response)

1 frames
/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py in __getattr__(self, name)
   1686             if name in modules:
   1687                 return modules[name]
-> 1688         raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")
   1689 
   1690     def __setattr__(self, name: str, value: Union[Tensor, 'Module']) -> None:

AttributeError: 'LlavaForConditionalGeneration' object has no attribute 'complete'

kkev

^ after loading from HF Transformers

LLogan M

Ah yea, we dont have mutlimodal huggingface support

LLogan M

at the moment

LLogan M

(it really is easier to use ollama though, huggingface will require a ton of config)

kkev

got it.

kkev

Plain Text

---------------------------------------------------------------------------
ConnectError                              Traceback (most recent call last)
/usr/local/lib/python3.10/dist-packages/httpx/_transports/default.py in map_httpcore_exceptions()
     65 
---> 66 @contextlib.contextmanager
     67 def map_httpcore_exceptions() -> typing.Iterator[None]:

22 frames
ConnectError: [Errno 99] Cannot assign requested address

The above exception was the direct cause of the following exception:

ConnectError                              Traceback (most recent call last)
/usr/local/lib/python3.10/dist-packages/httpx/_transports/default.py in map_httpcore_exceptions()
     81 
     82         if mapped_exc is None:  # pragma: no cover
---> 83             raise
     84 
     85         message = str(exc)

ConnectError: [Errno 99] Cannot assign requested address

kkev

i get this when trying to run a query on image iterable object.

response = mm_program(query_str="xxxxx?")
print(response)

kkev

mm_program is <llama_index.core.program.multi_modal_llm_program.MultiModalLLMCompletionProgram at 0x7e35985700d0>

LLogan M

mmmm no idea on that one, pretty sus error

LLogan M

never seen that before

kkev

yea, i can't tell from the error what's going on either / where to look

kkev

I am using ollama

kkev

seems to be something to with my http connection in my python environment. I am running this on google colab, if that helps.

kkev

do you think it could be related to ollama?

OllamaMultiModal(base_url='http://localhost:11434', model='llava:13b', temperature=0.75, context_window=3900, request_timeout=None, additional_kwargs={})

kkev

Ollama multimodal base_url is pointing to localhost, I am not sure how that would work on colab.

kkev

Plain Text

curl -fsSL https://ollama.com/install.sh | sh
>>> Downloading ollama...
####################################################################################################################################### 100.0%####################################################################################################################################### 100.0%
>>> Installing ollama to /usr/local/bin...
>>> Creating ollama user...
>>> Adding ollama user to video group...
>>> Adding current user to ollama group...
>>> Creating ollama systemd service...
WARNING: Unable to detect NVIDIA/AMD GPU. Install lspci or lshw to automatically detect and install GPU dependencies.
>>> The Ollama API is now available at 127.0.0.1:11434.
>>> Install complete. Run "ollama" from the command line.

kkev

this worked ^

Add a reply

Find answers from the community

Is there a good example / cookbook for