Find answers from the community

N
Nived
Offline, last seen 2 weeks ago
Joined January 23, 2025
Hello there, was wondering if we could pass images in prompt template

currently what i am doing is
prompt = prompt_template.format(
system_prompt=agent.system_prompt,
history=message_history,
context=context_str,
question=query,
agent_id=str(agent.id),
)

openai_agent = OpenAIAgent.from_tools(
tools_to_use,
llm=llm,
verbose=True,
tool_call_parser=advanced_tool_call_parser,
)
response = openai_agent.stream_chat(prompt)

but I want to add image reasoning also. was wondering how we do it. If anyone have idea, please help
I am able to get response like doing this
sample_message = ChatMessage(
role=MessageRole.USER,
blocks=[
TextBlock(text=text),
ImageBlock(url=image_url)
],
)

openai_llm = OpenAI(model=settings.OPENAI_MODEL, api_key=settings.OPEN_AI_API)
re = openai_llm.chat(messages=[sample_message])

but having difficulty integrating to stream_chat and prompt template
3 comments
L
N
N
Nived
·

Image

Hi guys, I am trying to do image reasoning with openai in llama-index. But the 'Image Block' is not being recognized. Can you please help me with this

llama-index version : 0.12.11
Python version: 3.13.1

from llama_index.llms.openai import OpenAI
from llama_index.core.llms import (
ChatMessage,
ImageBlock,
TextBlock,
MessageRole,
)

msg = ChatMessage(
role=MessageRole.USER,
blocks=[
TextBlock(text="Describe the images as an alternative text"),
ImageBlock(url="https://letsenhance.io/static/8f5e523ee6b2479e26ecc91b9c25261e/1015f/MainAfter.jpg"),
],
)
openai_llm = OpenAI(model="gpt-4o", max_new_tokens=300)
response = openai_llm.chat(messages=[msg])

this is the block I have been trying

response = openai_llm.chat(messages=[msg])
DEBUG:httpx:load_ssl_context verify=True cert=None trust_env=True http2=False
DEBUG:httpx:load_verify_locations cafile='/Users/nivedhari/agents/.venv/lib/python3.13/site-packages/certifi/cacert.pem'
DEBUG:openai._base_client:Request options: {'method': 'post', 'url': '/chat/completions', 'files': None, 'json_data': {'messages': [{'role': 'user', 'content': 'Describe the images as an alternative text'}], 'model': 'gpt-4o', 'stream': False, 'temperature': 0.1}}
DEBUG:openai._base_client:Sending HTTP Request: POST https://api.openai.com/v1/chat/completions
DEBUG:httpcore.connection:connect_tcp.started host='api.openai.com' port=443 local_address=None timeout=60.0 socket_options=None

this is the logs of the issue. The Image url is not being included in the content. anyone came across the same issue? Please help
2 comments
L
N