Does query engine come with retry

cchantlong

Does query engine come with retry mechanism? I am getting Request Timeout when calling Azure OpenAI.

8 comments

LLogan M

The openai client has retries built in, but you might just need to increase the timeout

cchantlong

Is there a way to log out if openai did a retry or not?

cchantlong

I get this trace. I'm guessing it's not even calling the LLM.

Plain Text

  | **********
  | Trace: query
  |     |_query ->  367.368937 seconds
  |       |_retrieve ->  0.94417 seconds
  |         |_embedding ->  0.191088 seconds
  |       |_synthesize ->  366.424662 seconds
  |         |_templating ->  6e-06 seconds
  |         |_llm ->  0.0 seconds
  | **********
  | **********
  | Trace: query
  |     |_query ->  367.368937 seconds
  |       |_retrieve ->  0.94417 seconds
  |         |_embedding ->  0.191088 seconds
  |       |_synthesize ->  366.424662 seconds
  |         |_templating ->  6e-06 seconds
  |         |_llm ->  0.0 seconds
  | **********

cchantlong

sorry I mean the synthesize part is lagging.

LLogan M

I think if you set the logging to debug you can see what openai is doing

LLogan M

Plain Text

import logging
import sys

logging.basicConfig(stream=sys.stdout, level=logging.DEBUG)
logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))

cchantlong

oh nice. thanks. I'm guessing that's more detailed than llama index set handler simple.

LLogan M

yea, that will show the debug logs from openais client

Add a reply

Find answers from the community

Does query engine come with retry