Find answers from the community

Updated 8 months ago

@Logan M I am trying to hook open

@Logan M I am trying to hook open telemetry into the tracing callback handler, but the context fetching or getting of the current span isnt working, is there a way to pass the context in or get it from your knowledge?
1
L
m
a
39 comments
what are you missing?
so 2 things:
  1. https://pypi.org/project/opentelemetry-instrumentation-llamaindex/ doesnt seem to work at all idk if you have any context on it
  2. if we start a span in the on_event_start it doesnt seem to work
Can you give an example? (Tagging in the author of this code, cc @nerdai)
just trying to picture what you mean, it works fine as far as I know lol
the demo notebook runs fine
πŸ€” Thanks for tagging. Oh this is using opentelemetry callback integration versus our new instrumentation module.
Actually this package is not managed by us.
ohhhhh good catch
I know arize uses open-tellemetry
I think this is likely their package
Yeah I remember them saying that in our last meeting with them!
im using the new tracing
but whats happening is if i create a new span in on start it gets killed once on_start ends
then when on_end happens it dies
is there a way for me to pass the span to on_end from on_start via a kwarg or something?
Plain Text
from time import sleep
from typing import Any, Dict, List, Optional
import uuid
import os
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import ConsoleSpanExporter, BatchSpanProcessor
from opentelemetry.sdk.trace.sampling import TraceIdRatioBased
from llama_index.core.callbacks.base_handler import BaseCallbackHandler
from llama_index.core.callbacks import (
    CallbackManager,
    LlamaDebugHandler,
)
from llama_index.core import Settings
from llama_index.core.callbacks.schema import (
    CBEventType,
)
from opentelemetry import trace
from opentelemetry.context import Context
from opentelemetry.trace import set_span_in_context
from opentelemetry.trace.span import Span
from opentelemetry.context import get_current
from llama_indexer.utils.traces import LLAMA_INDEXER_TRACER_NAME
from opentelemetry.context import attach


IS_DEV = os.getenv("PYTHON_ENV", "production") == "development"


class OpenTelemetryCallbackHandler(BaseCallbackHandler):
    def __init__(
        self,
        event_starts_to_ignore: List[CBEventType],
        event_ends_to_ignore: List[CBEventType],
    ):
        super().__init__(event_starts_to_ignore, event_ends_to_ignore)
        self.span_map: Dict[str, Span] = {}

    def on_event_start(
        self,
        event_type: CBEventType,
        payload: Optional[Dict[str, Any]] = None,
        event_id: str = "",
        parent_id: str = "",
        **kwargs: Any,
    ) -> str:
        print(f"Event start: {event_type}")
        if event_type in self.event_starts_to_ignore:
            return event_id

        tracer = trace.get_tracer(LLAMA_INDEXER_TRACER_NAME)

        current_span = trace.get_current_span()
        print(f"Current Span: {current_span}")
        new_span = trace.get_tracer(__name__).start_span(
            event_type.name,
            context=trace.set_span_in_context(current_span),
            kind=trace.SpanKind.INTERNAL,
        )

        context = get_current()
        context = trace.set_span_in_context(new_span, context)

        print(f"New Span: {new_span}")

        return event_id

    def on_event_end(
        self,
        event_type: CBEventType,
        payload: Optional[Dict[str, Any]] = None,
        event_id: str = "",
        **kwargs: Any,
    ) -> None:
        print(f"Event end: {event_type}")
        if event_type in self.event_ends_to_ignore:
            return
        span = trace.get_current_span()
        print(f"Span end: {span}")
        if span:
            if payload and IS_DEV:
                span.set_attribute("payload", str(payload))
            span.end()

    def start_trace(self, trace_id: Optional[str] = None) -> None:
        return None

    def end_trace(
        self,
        trace_id: Optional[str] = None,
        trace_map: Optional[Dict[str, List[str]]] = None,
    ) -> None:
        return None

    def flush(self):
        self.trace_id = None


handlers: List[BaseCallbackHandler] = [
    OpenTelemetryCallbackHandler([CBEventType.TEMPLATING], [CBEventType.TEMPLATING])
]
if IS_DEV:
    handlers.append(LlamaDebugHandler(print_trace_on_end=True))

Settings.callback_manager = CallbackManager(handlers)


def get_callback_manager() -> CallbackManager:
    return Settings.callback_manager
this is the sort of thing we are doing here
but the trace doesnt persist afer on_event_start no matter what we do
we have tried a. TON of things with no luck
ohhh this is using the old callback stuff, not the new instrumentation library
link to the docs por favor i thought this was that new stuff
(which is fine!)
IDC as long as it works hahaha!
what is the end goal here? Like, what data are you trying to collect?

Because there is this
https://docs.llamaindex.ai/en/stable/examples/instrumentation/basic_usage/
we just want to track all llm calls and internal llama index actions as spans
this looks much closer to what we need
Try it out! Thats the new thing πŸ™‚
trying it, getting a lot of type errors even just using the code given in documentation but im gonna see if it works
don't let the type errors scare you lol
I should note this was added in v0.10.20
What we're trying to do is use this to hook up Opentelemetry spans at each of the callback events. From what we can tell there's no built in support for that so either a) we need to implement our own callback handler (old way apparently), or b) implement an event or span handler (new way). A couple questions for you though
1) It looks like in the new span handler, it takes a generic span parameter which doesn't seem like it needs to extend BaseSpan (I'll try this)
2) Main concern I have is whether this is thread safe. From what I can tell it isn't. There's a coment in the code: " # TODO: thread safe?", also, it's setting the "current_span" when each span starts, so it doesn't really look like this can work as a global span handler since it would get confused with multiple concurrent spans. Do you have a thread safe solution for spans in a production environment? I'm not even sure how I'd set a request local handler for each request since it would be a lot of plumbing to pass that around.
Oh and one more question: Where is it controlled (or how do you control) which events case spans to get created?
events don't create spans, spans are created, and events happen within a span

basically, there is a start/drop span function, or a function decorator
https://docs.llamaindex.ai/en/stable/module_guides/observability/instrumentation/#enteringexiting-a-span
Ok, so I can just look for the places that have the @dispatcher.span decorator and those are the methods that have built in llamaindex spans?
What about the thread safety question above?
Yea exactly! You can even decorate functions outside of llama-index if you want
Bit of a question mark at the moment, but its being worked on. I might just give it a try and see what happens lol
Well reading through the code it's pretty clear it isn't threadsafe. It uses "current_span_id" to determine the parent so if there are concurrent requests that current_span_id will get overwritten
Add a reply
Sign up and join the conversation on Discord