Find answers from the community

Updated 2 months ago

Plans to integrate anthropics with other models for faster computation

Any plans of doing anthropics rag context with other models. Its so slow bc i tier 1 for anothropic

7 comments

You can technically follow the cookbook and use any model

But anthropic makes it so viable because of cheap prompt caching

OpenAI will automatically prompt cache as well now, but its only a 50% price reduction

Oh dang will try wait how does it work with local models assumin i'm renting GPU lmao

extra_headers={"anthropic-beta": "prompt-caching-2024-07-31"},
do we need to change this to anything

Thats just specific to using anthropic

local models will probably be slower lol I would use vLLM or TGI to host

Add a reply