I have a strange issue - trying to get a

RRunonthespot

I have a strange issue - trying to get a llama-index working with Azure. I've managed to get embedding working fine, and when I run the query, I get the nodes from the query fine, but at the point that I get a streaming response, I get an error from llms\openai.py (which is already odd, as I thought it should be using AzureOpenai.py?)

if (delta.role == MessageRole.ASSISTANT) and (delta.content is None):
AttributeError: 'dict' object has no attribute 'role'

I'm using
llm = AzureOpenAI ( ...)
embed_model = AzureOpenAIEmbeddings(... )

service_context = ServiceContext.from_defaults(embed_model=embed_model, llm=llm)
...
response = query_engine.query(query)

for text in response.response_gen:
....

I get back a response fine and response.source_nodes returns some nodes, but I can't seem to iterate the reponse_gen.

21 comments

LLogan M

AzureOpenAI is just a light wrapper around the OpenAI LLM, so it makes sense that's where the error is (openai.py implements all almost the logic)

We recently updated to the new OpenAI client -- it should be returning pydantic objects, not dicts 🤔

If you run pip show openai what do you get?

LLogan M

actually nvm it would error out way earlier if you had the wrong openai version

LLogan M

does a non-streaming query work?

RRunonthespot

It's Version 1.2.3 as required I think by llama-index latest/greatest. I think they've (openai) made a tonne of changes since my app has regular retriever-less non-llamaindex chat and I had to do things like convert the old:
for chunk in response:
...
content = json.dumps(chunk)

which changed to
content = chunk.model_dump_json()

since the response was returning pydantic objects/

So you reckon this might be an incompatability between openai's latest python library and the Azure services?

I don't see AzureOpenAI.py in the call tree which is what I'd expect.
Some additional clues:
-Am using CitationQueryEngine (if that makes any difference?)

Will attempt a non-streaming response but it's a bit of butchery to make it do that as this sits behind a fastapi / called from a react app

RRunonthespot

...
\venv\Lib\site-packages\starlette\routing.py app 69
\venv\Lib\site-packages\starlette\responses.py call 270
\venv\Lib\site-packages\starlette\responses.py wrap 273
\venv\Lib\site-packages\starlette\responses.py stream_response 262
\Nova.API\nova\main.py stream_response 118
\venv\Lib\site-packages\llama_index\llm_predictor\utils.py gen 28
\venv\Lib\site-packages\llama_index\llms\base.py wrapped_gen 193
\venv\Lib\site-packages\llama_index\llms\openai.py gen 306

RRunonthespot

e.g. stream_response (little func that is iterating the returned response.response_gen seems to be hitting openai.py not AzureOpenAI.py that I'd expect

RRunonthespot

Index flow

Attachment

RRunonthespot

query flow

Attachment

LLogan M

Yea as o mentioned, AzureOpenAI extends openai, so that makes sense

https://github.com/run-llama/llama_index/blob/29ef306ae0536de44840ca5acfdf93d84b9a560c/llama_index/llms/azure_openai.py#L16

We've made a ton of changes to accommodate the new openai client version. But it seems to be breaking its own rules by returning a dict for delta

LLogan M

Very sus, I don't have azure to debug with at the moment 🤔

RRunonthespot

So something I noticed with azure that is different to openai- it does return the dicts but sometimes there is a precursor message with the content filter stuff that doesn’t necessarily have the delta or content included. I had to put in checks to make sure that only process the streaming back packet if it actually has those keys

RRunonthespot

So in my chat thing I do this

Attachment