Find answers from the community

Updated 2 months ago

Plans to integrate anthropics with other models for faster computation

Any plans of doing anthropics rag context with other models. Its so slow bc i tier 1 for anothropic
L
d
7 comments
You can technically follow the cookbook and use any model
But anthropic makes it so viable because of cheap prompt caching
OpenAI will automatically prompt cache as well now, but its only a 50% price reduction
Oh dang will try wait how does it work with local models assumin i'm renting GPU lmao
extra_headers={"anthropic-beta": "prompt-caching-2024-07-31"},
do we need to change this to anything
Thats just specific to using anthropic
local models will probably be slower lol I would use vLLM or TGI to host
Add a reply
Sign up and join the conversation on Discord