Find answers from the community

Updated last year

it requests for ```from llama_index.

At a glance
it requests for
Plain Text
from llama_index.multi_modal_llms.replicate_multi_modal import (
    REPLICATE_MULTI_MODAL_LLM_MODELS,
)
L
a
r
22 comments
Remove the import, update code

Plain Text
            llava_multi_modal_llm = ReplicateMultiModal(
                model="yorickvp/llava-13b:2facb4a474a0462c15041b78b1ad70952ea46b5ec6ad29583c0b29dbd4249591",
                max_new_tokens=100,
                temperature=0.1,
                num_input_files=1,
                top_p=0.9,
                num_beams=1,
                repetition_penalty=1,
            )
Plain Text
from llama_index.multi_modal_llms import ReplicateMultiModal
from llama_index.multi_modal_llms.replicate_multi_modal import (
    REPLICATE_MULTI_MODAL_LLM_MODELS,
)
    llava_multi_modal_llm = ReplicateMultiModal(
                model="yorickvp/llava-13b:2facb4a474a0462c15041b78b1ad70952ea46b5ec6ad29583c0b29dbd4249591",
                max_new_tokens=100,
                temperature=0.1,
                num_input_files=1,
                top_p=0.9,
                num_beams=1,
                repetition_penalty=1,
            )

prompts = [
    "what is shown in this image?",
    "how many people are shown in the image?",
    "is there anything unusual in the image?",
]
is this correct?
yea looks fine
i get error
Plain Text
ImportError                               Traceback (most recent call last)
<ipython-input-84-3278bba7d5e7> in <cell line: 2>()
      1 from llama_index.multi_modal_llms import ReplicateMultiModal
----> 2 from llama_index.multi_modal_llms.replicate_multi_modal import (
      3     REPLICATE_MULTI_MODAL_LLM_MODELS,
      4 )
      5 llava_multi_modal_llm = ReplicateMultiModal(

ImportError: cannot import name 'REPLICATE_MULTI_MODAL_LLM_MODELS' from 'llama_index.multi_modal_llms.replicate_multi_modal' (/usr/local/lib/python3.10/dist-packages/llama_index/multi_modal_llms/replicate_multi_modal.py)

---------------------------------------------------------------------------
NOTE: If your import is failing due to a missing package, you can
manually install dependencies using either !pip or !apt.

To view examples of installing some common dependencies, click the
"Open Examples" button below.
---------------------------------------------------------------------------
remove the second import then, it's not used (although weird πŸ€·β€β™‚οΈ )
but it is also used in
Plain Text
res = []
for prompt_idx, prompt in enumerate(prompts):
    for image_idx, image_doc in enumerate(image_documents):
        for llm_idx, llm_model in enumerate(REPLICATE_MULTI_MODAL_LLM_MODELS):
            ## Initialize the MultiModal LLM model
            llava_multi_modal_llm = ReplicateMultiModal(
                model=REPLICATE_MULTI_MODAL_LLM_MODELS[llm_model],
                max_new_tokens=100,
                temperature=0.1,
                num_input_files=1,
                top_p=0.9,
                num_beams=1,
                repetition_penalty=1,
            )

            llava_resp = llava_multi_modal_llm.complete(
                prompt=prompt,
                image_documents=[image_doc],
            )
            res.append(
                {
                    "model": llm_model,
                    "prompt": prompt,
                    "response": llava_resp,
                    "image": str(image_doc.image_path),
                }
            )
Yea again, pass in the model name/url that you actually want to use
what to use for : REPLICATE_MULTI_MODAL_LLM_MODELS
Plain Text
ImportError: cannot import name 'REPLICATE_MULTI_MODAL_LLM_MODELS' from 'llama_index.multi_modal_llms.replicate_multi_modal' (/usr/local/lib/python3.10/dist-packages/llama_index/multi_modal_llms/replicate_multi_modal.py)
It's not realeased yet I guess on pypi, it's still in the main branch

Just don't import it, use this instead

Plain Text
REPLICATE_MULTI_MODAL_LLM_MODELS = {
    "llava-13b": "yorickvp/llava-13b:2facb4a474a0462c15041b78b1ad70952ea46b5ec6ad29583c0b29dbd4249591",
    "fuyu-8b": "lucataco/fuyu-8b:42f23bc876570a46f5a90737086fbc4c3f79dd11753a28eaa39544dd391815e9",
    "minigpt-4": "daanelson/minigpt-4:b96a2f33cc8e4b0aa23eacfce731b9c41a7d9466d9ed4e167375587b54db9423",
}
thanks , i got another error:
Plain Text
res = []
for prompt_idx, prompt in enumerate(prompts):
    for image_idx, image_doc in enumerate(image_documents):
        for llm_idx, llm_model in enumerate(REPLICATE_MULTI_MODAL_LLM_MODELS):
            ## Initialize the MultiModal LLM model
            llava_multi_modal_llm = ReplicateMultiModal(
                model=REPLICATE_MULTI_MODAL_LLM_MODELS[llm_model],
                max_new_tokens=100,
                temperature=0.1,
                num_input_files=1
            )

            llava_resp = llava_multi_modal_llm.complete(
                prompt=prompt,
                image_documents=[image_doc],
            )
            res.append(
                {
                    "model": llm_model,
                    "prompt": prompt,
                    "response": llava_resp,
                    "image": str(image_doc.image_path),
                }
            )
gives error:
Plain Text
ModelError                                Traceback (most recent call last)
<ipython-input-92-bebb2169bb99> in <cell line: 2>()
     11             )
     12 
---> 13             llava_resp = llava_multi_modal_llm.complete(
     14                 prompt=prompt,
     15                 image_documents=[image_doc],

3 frames
/usr/local/lib/python3.10/dist-packages/replicate/run.py in run(client, ref, input, **params)
     56 
     57     if prediction.status == "failed":
---> 58         raise ModelError(prediction.error)
     59 
     60     return prediction.output

ModelError: probability tensor contains either `inf`, `nan` or element < 0
i am using Langchain to pull the text and tables as images from the pdf/ these are images i am using to run the code @Logan M
i got 3 tables and rest of it is text ... so now using only the images and applying llama index
@Logan M are you there?
@Logan M Good Morning, Any updates on this ticket: https://github.com/run-llama/llama_index/issues/9146
@andysingal are you still facing the issue? I think its because of the format of images you are using.
yes, i do... i am getting .jpg files which includes images of tables, charts. In what format do the images need to be... i am extracting images, tables from attached pdf
Attachment
Screenshot_2023-11-27_at_10.08.26_AM.png
Add a reply
Sign up and join the conversation on Discord