LlamaIndex

Log inLog into community

Find answers from the community

Updated 2 years ago

even with debug logging enabled

even with debug logging enabled

At a glance

The community member is having an issue with the llamaindex library, where even with debug logging enabled, it does not print out the queries or information being sent via the service context. The community members discuss potential solutions, such as using a callback like the TokenCountingHandler, which can track LLM input and output. However, they encounter issues with the callback not firing due to the llamaindex library returning "garbage" and causing the LLM selector prompt to fail. The community members suggest wrapping the query in a try/except block to log the exceptions and tracebacks, which may provide more insight into the issue.

Useful resources

·

even with debug logging enabled, llamaindex doesn't print out the queries/information that it's sending via the service context. Is there any way to make it log that stuff?

L

t

36 comments

You'll have to use a callback at the point

The easiest to use will be the token counting handler -- it keep track of every LLM input and output (example at the end)
https://gpt-index.readthedocs.io/en/stable/examples/callbacks/TokenCountingHandler.html

woof

that's unpleasant

🙂

nah it's fine

programatic access to logs is nice

looking at the TokenCountingHandler, I see:

Plain Text

token_counter = TokenCountingHandler(
    tokenizer=tiktoken.encoding_for_model("gpt-3.5-turbo").encode
)

It's not clear whether or not this is an embedding model situation. Would it be possible to use encoding_for_model("local:BAAI/bge-base-en") here?

Would not be possible -- tbh this is only useful for when counting tokens matters (i.e APIs that charge per token)

You can exclude that argument for now

If you actually wanted to properly count tokens here, you would have to pass in a llama2 tokenizer from huggingface

i don't want to count tokens

i'm just looking at the docs

Then you can leave that argument out, it will default to something else

so I can just do TokenCountingHandler() ?

yessir

(i think)

so unfortunately this doesn't work for a different reason you didn't expect

the callback only fires if the "transaction" is successful

but because llama2 often returns garbage, the llm selector prompt blows up with a traceback, so the callback never fires

while it was unconscious, in hindsight this is exactly the reason why the callback-type logging isn't sufficient

in the process of debugging what's going on, callbacks don't help, and we're back to: even with debug logging turned on, it doesn't spit out the right things

But it should be logged before it blows up -- wrap the query in a try/except and you should still be able to see the logged llm calls in the handler

It gets logged at the lowest level possible

sorry, sounds like streams crossed here --

debug logging doesn't log anything related to the calls to the llm.

I'm not sure how try/except would fix that

initially i said "debug logging doesn't log any calls to the llm" and you said "yes, you have to use callbacks"

are you saying that wrapping query_engine.query in a try/except will result in the callback successfully firing, even though the query fails?

Sorry, I was tallying about logging to the callback handler

Yes

in try/except is there a way to print the exception that ocurred?

and/or the traceback

Yea

Plain Text

try:
  ...
except Exception as e:
  print(e)

There's a way to print the full trace too, that's just the root error/exception

I gotta look it up haha

🙂

Updated example that prints the full trace too

Plain Text

import traceback

try:
  ...
except Exception:
  traceback.print_exc()

👍

Add a reply

Sign up and join the conversation on Discord