When we instrument LlamaIndex for

ttimothybeamish

When we instrument LlamaIndex for observability, we start to see the output of @dispatcher.span decorators that are already in the framework code. Most of the time, this is great. Is it possible to filter which of these we actually want in our final OpenTel logs?

In our specific case, the BaseRetriever class has a @dispatcher.span decorator. Meanwhile, we have one on each of our child retrievers. The result is an OpenTel trace that shows:
-- BaseRetriever
---- CustomRetriever

where we don't really need the top BaseRetriever log so I'm looking for a way to filter it out.

2 comments

LLogan M

Hmmm, I think you'd have to override the dispatchers span_enter() method to filter, and then call super() ?

Plain Text

dispatcher = get_dispatcher()
dispatcher.span_enter = my_span_enter_fn

Not 100% sure if pydantic will let you do that though (since the root dispatcher is both a singleton and a BaseModel)

Or if you have a span handler, could filter there potentially, but looking at the code that was less clear

FFatherMonkey

Hey there!
It looks like you're dealing with some clutter in your OpenTel logs due to the spans created by the BaseRetriever class. One of the main challenges here is filtering out those unnecessary top-level logs while keeping the relevant child retriever spans. A practical solution would be to customize your logging configuration to exclude spans from the BaseRetriever class by setting up a filter in your OpenTelemetry setup. This way, you can focus on the spans that matter most to you, like those from your custom retrievers. I’d be happy to help you implement this filtering to streamline your observability!
Best

Add a reply

Find answers from the community

When we instrument LlamaIndex for