Hi there,

At a glance

Hi there,
I’m using a local embedding model and Azure open AI for the response synthesis. I’m getting response times of about 10s as measured by:

Plain Text

start_time = time.time()
response = query_engine.query(question)
response_time = time.time()-start_time

Is it possible to get execution times of what happens inside this (black for me) box query_engine.query()? I need to know if the 10s are mostly due to azure open AI (and thus I can’t do anything about it) or if they are coming from the local embedding on my machine.

For instance, does OpenAI return the time it took to create a response on its server?

8 comments

TTeemu

Do you have this setup?

Plain Text

import logging
import sys

logging.basicConfig(stream=sys.stdout, level=logging.DEBUG)
logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))

There are also other tools that might be able to help: https://docs.llamaindex.ai/en/stable/understanding/tracing_and_debugging/tracing_and_debugging/

SStéphane

now I have it 🙂 although I don’t see any obvious execution time or timestamp in the large amount of text I see.

WWhiteFang_Jr

You can use other more powerful tools as @Teemu mentioned for more granular details about each step:
for example take a look at Langfuse: https://docs.llamaindex.ai/en/stable/examples/callbacks/LangfuseCallbackHandler/?h=lang

LLogan M

Its almost certainly the LLM

You can actually run all the steps the query engine is doing yourself too

For example

Plain Text

from llama_index.core import get_response_synthesizer

index = ...

retriever = index.as_retriever(similarity_top_k=2)

synth = get_response_synthesizer(llm=llm, response_mode="compact")

nodes = retriever.retrieve(query_str)

response = synth.synthesize(query_str, nodes)

LLogan M

That is what the query engine does (if you had any node-postprocessors, those get applied between the retrieval and synthesis)

jjuliendemarque

@Stéphane Hey! Qu'est ce que tu fais avec llamaindex? Ton livre est top btw.

SStéphane

@juliendemarque GourouFabulus, le chatbot qui répondra à toutes vos questions sur la psycho évo !

jjuliendemarque

@Stéphane ohh ahahah nice.

Add a reply

Find answers from the community

Hi there,