Cached Augmented Generation (CAG) with Gemini or other LLMs integrated with LlamaIndex

correct me if I'm wrong, but doesn't CAG require direct model access (i.e. with pytorch)? I don't think you can implement this over an API

You can see an example here: https://github.com/hhhuang/CAG/blob/main/kvcache.py