LlamaIndex

Log inLog into community

Find answers from the community

Updated 9 months ago

Is there a good example / cookbook for

Is there a good example / cookbook for

At a glance

The community members are looking for a good example or cookbook for using multi-vector/recursive retriever and multi-modal RAG with the llama-index library. They have found a LangChain example, and the comments suggest checking the llama-index documentation for multimodal use cases and examples.

The community members also discuss trying to use the llava model from Hugging Face instead of the ollama model, but encounter issues with the LlavaForConditionalGeneration model not having the required complete attribute. They note that Hugging Face multimodal support is not available at the moment, and it would require significant configuration, so it's easier to use the ollama model.

The community members also encounter a ConnectError when trying to run a query on an image iterable object, which they suspect may be related to the ollama multimodal setup and the fact that they are running on Google Colab. They provide steps to install the ollama API locally, which seems to have resolved the issue.

Useful resources

·

Is there a good example / cookbook for multi-vector / recursive retriever + multi-modal RAG using llama-index?

Here's an LangChain example: https://github.com/langchain-ai/langchain/blob/master/cookbook/Multi_modal_RAG.ipynb

L

k

21 comments

Have you seen this page? https://docs.llamaindex.ai/en/stable/use_cases/multimodal/

This is pretty comprehensive for example
https://docs.llamaindex.ai/en/stable/examples/multi_modal/ollama_cookbook/?h=multimodal

@Logan M I was hoping to load the llava model from HF instead of ollama . any idea how it would work in this example? https://docs.llamaindex.ai/en/stable/examples/multi_modal/ollama_cookbook/?h=multimodal

doesn't look like I can simply pass the model from HF to multi_modal_llm parameter.

Plain Text

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-58-6d30c419b9a0> in <cell line: 1>()
----> 1 response = mm_program(query_str="What was the value of Non Revenue Units in Apr 2022?")
      2 print(response)

1 frames
/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py in __getattr__(self, name)
   1686             if name in modules:
   1687                 return modules[name]
-> 1688         raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")
   1689 
   1690     def __setattr__(self, name: str, value: Union[Tensor, 'Module']) -> None:

AttributeError: 'LlavaForConditionalGeneration' object has no attribute 'complete'

^ after loading from HF Transformers

Ah yea, we dont have mutlimodal huggingface support

at the moment

(it really is easier to use ollama though, huggingface will require a ton of config)

got it.

Plain Text

---------------------------------------------------------------------------
ConnectError                              Traceback (most recent call last)
/usr/local/lib/python3.10/dist-packages/httpx/_transports/default.py in map_httpcore_exceptions()
     65 
---> 66 @contextlib.contextmanager
     67 def map_httpcore_exceptions() -> typing.Iterator[None]:

22 frames
ConnectError: [Errno 99] Cannot assign requested address

The above exception was the direct cause of the following exception:

ConnectError                              Traceback (most recent call last)
/usr/local/lib/python3.10/dist-packages/httpx/_transports/default.py in map_httpcore_exceptions()
     81 
     82         if mapped_exc is None:  # pragma: no cover
---> 83             raise
     84 
     85         message = str(exc)

ConnectError: [Errno 99] Cannot assign requested address

i get this when trying to run a query on image iterable object.

response = mm_program(query_str="xxxxx?")
print(response)

mm_program is <llama_index.core.program.multi_modal_llm_program.MultiModalLLMCompletionProgram at 0x7e35985700d0>

mmmm no idea on that one, pretty sus error

never seen that before

yea, i can't tell from the error what's going on either / where to look

I am using ollama

seems to be something to with my http connection in my python environment. I am running this on google colab, if that helps.

do you think it could be related to ollama?

OllamaMultiModal(base_url='http://localhost:11434', model='llava:13b', temperature=0.75, context_window=3900, request_timeout=None, additional_kwargs={})

Ollama multimodal base_url is pointing to localhost, I am not sure how that would work on colab.

Plain Text

curl -fsSL https://ollama.com/install.sh | sh
>>> Downloading ollama...
####################################################################################################################################### 100.0%####################################################################################################################################### 100.0%
>>> Installing ollama to /usr/local/bin...
>>> Creating ollama user...
>>> Adding ollama user to video group...
>>> Adding current user to ollama group...
>>> Creating ollama systemd service...
WARNING: Unable to detect NVIDIA/AMD GPU. Install lspci or lshw to automatically detect and install GPU dependencies.
>>> The Ollama API is now available at 127.0.0.1:11434.
>>> Install complete. Run "ollama" from the command line.

this worked ^

Add a reply

Sign up and join the conversation on Discord