Does query engine come with retry

At a glance

The community member is experiencing a "Request Timeout" issue when calling Azure OpenAI. Other community members suggest that the OpenAI client has retries built-in, and the timeout may need to be increased. They also recommend enabling debug logging to see more details on what the OpenAI client is doing. The community members discuss the possibility of logging whether OpenAI retried the request, and they analyze a trace that shows the "synthesize" part of the query is taking a long time, but the LLM is not being called.

cchantlong

Does query engine come with retry mechanism? I am getting Request Timeout when calling Azure OpenAI.

8 comments

LLogan M

The openai client has retries built in, but you might just need to increase the timeout

cchantlong

Is there a way to log out if openai did a retry or not?

cchantlong

I get this trace. I'm guessing it's not even calling the LLM.

Plain Text

  | **********
  | Trace: query
  |     |_query ->  367.368937 seconds
  |       |_retrieve ->  0.94417 seconds
  |         |_embedding ->  0.191088 seconds
  |       |_synthesize ->  366.424662 seconds
  |         |_templating ->  6e-06 seconds
  |         |_llm ->  0.0 seconds
  | **********
  | **********
  | Trace: query
  |     |_query ->  367.368937 seconds
  |       |_retrieve ->  0.94417 seconds
  |         |_embedding ->  0.191088 seconds
  |       |_synthesize ->  366.424662 seconds
  |         |_templating ->  6e-06 seconds
  |         |_llm ->  0.0 seconds
  | **********

cchantlong

sorry I mean the synthesize part is lagging.

LLogan M

I think if you set the logging to debug you can see what openai is doing

LLogan M

Plain Text

import logging
import sys

logging.basicConfig(stream=sys.stdout, level=logging.DEBUG)
logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))

cchantlong

oh nice. thanks. I'm guessing that's more detailed than llama index set handler simple.

LLogan M

yea, that will show the debug logs from openais client

Add a reply

Find answers from the community

Does query engine come with retry