If I set a callback handler like this,

bbryanvann

If I set a callback handler like this, it should be global right and apply to all embeddings and llm calls automatically? I tried it but it doesn't seem to be triggering the callback manager for some reason.

Plain Text

llama_debug = LlamaDebugHandler(print_trace_on_end=True)
    Settings.callback_manager = CallbackManager([llama_debug])

11 comments

LLogan M

Hmmm, try setting Settings.global_handler = llama_debug instead

LLogan M

There might be some missed stuff with the global callback manager :PSadge:

bbryanvann

Tried that but I think that's the old way that expects a string such as "simple". But it failed when I tried it that way so I think it expects you to use set_global_handler("simple")
according to the docs here https://docs.llamaindex.ai/en/stable/module_guides/observability/observability.html While "simple" will output llm calls it doesn't show the embeddings calls

LLogan M

ahhh you are correct

LLogan M

major sadge

LLogan M

what does you code look like beyond that? tbh it should be pulling the callback manager from the settings

LLogan M

theres a bunch of code I added to do that, but maybe I missed the component you are using

bbryanvann

This is roughly our code (simplified a bit). When I run this and then print the token_counter values for total llm token count and total embedding token count they're both 0. Also I never see any stack traces printed from the LlamaDebugHandler. I've tried it both ways where I pass in the callback manager and where I don't and the result is the same in either case. Could it be because we're using AzureOpenAI and AzureOpenAIEmbeddings rather than directly using OpenAI?

Plain Text

token_counter = TokenCountingHandler(
tokenizer=tiktoken.encoding_for_model("gpt-3.5-turbo").encode,
    verbose=True,
)

llama_debug = LlamaDebugHandler(print_trace_on_end=True)

callback_manager = Settings.callback_manager
callback_manager.add_handler(token_counter)
callback_manager.add_handler(llama_debug)

def get_worker_llm():
    return AzureOpenAI(model="gpt-3.5-turbo", ...)

def get_embed_model():
    return AzureOpenAIEmbedding(...)

def run_ingestion_transforms_on_documents(documents: List[Document]) -> List[BaseNode]:
    node_parser = SentenceSplitter(
        separator=" ",
        chunk_size=1024,
        chunk_overlap=200,
        # callback_manager=callback_manager,
    )

    parsed_nodes = node_parser.get_nodes_from_documents(documents)

    extractors: List[BaseExtractor] = [
        SummaryExtractor(summaries=["prev", "self", "next"], llm=get_worker_llm()),
        QuestionsAnsweredExtractor(questions=3, llm=get_worker_llm()),
    ]

    for extractor in extractors:
        parsed_nodes = extractor(
            parsed_nodes,
            #   callback_manager=callback_manager
        )

    embed_model = get_embed_model()
    embedded_nodes = embed_model(
        parsed_nodes,
        #   callback_manager=callback_manager
    )
    return embedded_nodes


def load_documents(path: str):
    documents = SimpleDirectoryReader(path).load_data()
    return documents

def ingest(path: str):
    documents = load_documents(path)
    run_ingestion_transforms_on_documents(documents)

LLogan M

Hmm, let me fiddle with this setup a bit

bbryanvann

Ok I think I finally figured this out. It started working once I added:

Plain Text

Settings.embed_model = get_embed_model()
Settings.llm = get_worker_llm()

It seems that if you just pass embed models or llms into the individual modules that it won't pick up the global handler. Still playing with it a bit to see if it's working as expected, but at least I can get events flowing through a custom handler I created now

LLogan M

ahhh interesting. Ok good to know!

Add a reply

Find answers from the community

If I set a callback handler like this,