This really helped me understand where Fine Tuning really shines and where it lacks, particularly in knowledge retention as we spoke of here.
https://arxiv.org/html/2312.05934v3Also, it seems that the fine tuning available via OpenAI's APIs really is only useful for very niche fine tuning on structure/persona/response formatting/chain of thought reasoning. You're unlikely to give it much additional knowledge although I am testing that myself for fun today.