is there a way to get this working with

Which client library ?

You can make a custom LLM class where you can implement this using ollama client

That would have to be a PR to use it under the hood instead of http requests

I wonder what the diffrence would be though?