Find answers from the community

s
F
Y
a
P
Updated 9 months ago

If I set a callback handler like this,

If I set a callback handler like this, it should be global right and apply to all embeddings and llm calls automatically? I tried it but it doesn't seem to be triggering the callback manager for some reason.

Plain Text
llama_debug = LlamaDebugHandler(print_trace_on_end=True)
    Settings.callback_manager = CallbackManager([llama_debug])
L
b
11 comments
Hmmm, try setting Settings.global_handler = llama_debug instead
There might be some missed stuff with the global callback manager :PSadge:
Tried that but I think that's the old way that expects a string such as "simple". But it failed when I tried it that way so I think it expects you to use set_global_handler("simple")
according to the docs here https://docs.llamaindex.ai/en/stable/module_guides/observability/observability.html While "simple" will output llm calls it doesn't show the embeddings calls
ahhh you are correct
what does you code look like beyond that? tbh it should be pulling the callback manager from the settings
theres a bunch of code I added to do that, but maybe I missed the component you are using
This is roughly our code (simplified a bit). When I run this and then print the token_counter values for total llm token count and total embedding token count they're both 0. Also I never see any stack traces printed from the LlamaDebugHandler. I've tried it both ways where I pass in the callback manager and where I don't and the result is the same in either case. Could it be because we're using AzureOpenAI and AzureOpenAIEmbeddings rather than directly using OpenAI?
Plain Text
token_counter = TokenCountingHandler(
tokenizer=tiktoken.encoding_for_model("gpt-3.5-turbo").encode,
    verbose=True,
)

llama_debug = LlamaDebugHandler(print_trace_on_end=True)

callback_manager = Settings.callback_manager
callback_manager.add_handler(token_counter)
callback_manager.add_handler(llama_debug)

def get_worker_llm():
    return AzureOpenAI(model="gpt-3.5-turbo", ...)

def get_embed_model():
    return AzureOpenAIEmbedding(...)

def run_ingestion_transforms_on_documents(documents: List[Document]) -> List[BaseNode]:
    node_parser = SentenceSplitter(
        separator=" ",
        chunk_size=1024,
        chunk_overlap=200,
        # callback_manager=callback_manager,
    )

    parsed_nodes = node_parser.get_nodes_from_documents(documents)

    extractors: List[BaseExtractor] = [
        SummaryExtractor(summaries=["prev", "self", "next"], llm=get_worker_llm()),
        QuestionsAnsweredExtractor(questions=3, llm=get_worker_llm()),
    ]

    for extractor in extractors:
        parsed_nodes = extractor(
            parsed_nodes,
            #   callback_manager=callback_manager
        )

    embed_model = get_embed_model()
    embedded_nodes = embed_model(
        parsed_nodes,
        #   callback_manager=callback_manager
    )
    return embedded_nodes


def load_documents(path: str):
    documents = SimpleDirectoryReader(path).load_data()
    return documents

def ingest(path: str):
    documents = load_documents(path)
    run_ingestion_transforms_on_documents(documents)
Hmm, let me fiddle with this setup a bit
Ok I think I finally figured this out. It started working once I added:
Plain Text
Settings.embed_model = get_embed_model()
Settings.llm = get_worker_llm()

It seems that if you just pass embed models or llms into the individual modules that it won't pick up the global handler. Still playing with it a bit to see if it's working as expected, but at least I can get events flowing through a custom handler I created now
ahhh interesting. Ok good to know!
Add a reply
Sign up and join the conversation on Discord