Find answers from the community

Updated 3 months ago

Hi Guys, I need to log token usage in a

Hi Guys, I need to log token usage in a concurrent fastapi app, I'm using both CallbackManager and TokenCountingHandler from llama_index.core.callbacks, but setting Settings.callback_manager is causing race conditions as Settings is a global state across the app, I also can see some classes which says the service context is deprecated so Settings is the way to go now, could someone please give me some light on how can I effectively log token counting in a concurrent app?

3 comments

LLogan M

llm = OpenAI(..., callback_manager=callback_manager)
embed_model = OpenAIEmbedding(..., callback_manager=callback_manager)

And then pass in the llm and embed model where they are needed

LLogan M

VectorStoreIndex(..., embed_model=embed_model)

.as_query_engine(llm=llm, ...)

etc.

ccoder.ve

Thank you 🤘

Add a reply