anyone have any ideas on this?

Not really anything built into the library.

These days though, DPO is much easier approach for fine-tuning compared to RLHF

Huggingface has quite a few utilities for this

got it. will look into DPO.

thanks

This might be a good article to start with 🙂 https://huggingface.co/blog/dpo-trl