I cannot figure out how to get a trace

hmmm I think the ingestion pipeline is not properly instrumented for callbacks 😅 Each element in the pipeline has a callback manager, but there is no overall trace points

@Logan M Would this constitute a potential feature? I think it would be a good idea to have this to get a better trace for the overall pipeline and simplify passing the callback_manager once instead of having to send it to individual elements.

Also, I think you cannot pass a callback_manager to a vector_store directly as well, so that also would not give me a complete trace even if I pass it to individual elements.

I think the ingestion pipeline overall should support having a callback manager. Definitely a potential feature

I don't nesscarily think the vector store needs one per-say, since that could be handled outside of the vector store. But maybe someday

Sounds good, should I create an issue on the repo to track this?

Might try implementing it as well, though it may take time 🙂