Callback

cchantlong

Question on the callback manager. Currently in my app, when I start it, I initialize it with Settings.callback_manager = callback_manager where callback_manager just has, llama_debug, tokencountinghandler in Django.

How can I import it correctly because when I try to use import Settings, and try to access Settings.callback_manager in a different file, it ignores what I setup globally and starts its own callback_manager.

Am I suppose to be starting an instance of token_counter per API request? If 2 API requests happen at the same time, I assume it'll conflict so... maybe I can't really use Settings.callback_manager globally? 🤔

4 comments

LLogan M

Try attaching it directly to your llm and embedding model, and passing those into the modules that are using them

For example

Plain Text

llm = OpenAI(..., callback_manager=callback_manager)
embed_model = OpenAIEmbedding(...  callback_manager=callback_manager)

index = VectorStoreIndex(..., embed_model=embed_model)

query_engine = index.as_query_engine(..., llm=llm)

cchantlong

thanks will try that.

cchantlong

actually, if I do it like that. how do I access the token counter and llama debug later?

LLogan M

The same way you did before? 👀

Add a reply

Find answers from the community

Callback