Context

Yea context_only and no-text don't call the llm

So there's no event to record

Wouldn't it make sense to record the QUERY event in that case?

But not sure how you would generalize that, because the on_event_start and on_event_end don't get the type of synthesizer used

Maybe, but the finetuning handler is recording events directly on the llm, and if the llm doesn't get called in these modes, there's nothing ever recorded

Just avoid those modes if you want to use the finetuning handler?

Maybe you can try to explain it to me, but how do the other modes even make sense?

My friend and myself contributed in developing the ContextOnly approach, because otherwise information just gets lost or no? The inner LLM never sees the chat history only the last question. The outer LLM never sees the context, only the answer of the inner LLM.

The ContextOnly approach solves this, because it gives the Context directly back to the outer LLM. We only have one LLM to consider and questions regarding older chat history doesn't become a problem anymore.

I tested this thoroughly and noticed it when using the basic ChatEngine. That's why i switched to the AgentRunner + ContextOnly mode.

That's also why i would like to use these modes - The other modes cause the problems.

Oh so you are using an agent on top, that makes more sense

I'm not really sure what the exact issue is then, I'd have to dive pretty deep

I wrote a custom OpenAIFinetuningHandler class that allows to handle the Agent messages by listening to the CBEventType.AGENT_STEP. You may consider this, when you want to make a PR there, i'm too lazy rn xD Don't know how relevant this is, and if you have many more tasks to do atm.

Now it works at least 🙂

Maybe my solution is a bit hacky though...