Prompt Caching with Claude

I'm very happy this is now available. With a model as smart as Claude 3.5, this opens up the possibility of writing prompts the size of relatively complex programs, prompts with many examples and targeting multiple outputs, and running them repeatedly on different datasets (big code, small data), at a reasonable cost. For example, a classifier with a hundred class labels described by prompts? A questionnaire that turns a piece of unstructured data into a long list of features? A complex custom assistant, with a long prompt listing individual user's available tools and preferences?

(my previous rants on why we need this: https://news.ycombinator.com/item?id=40034972#40036309 )

Now we just need to merge Anthropic's cache pricing with Google's context length (their cache pricing isn't as good) and OpenAI's strict structured output mode...

This is amazing! You could easily have your RAG app return the top 100 results now. You could probably do 1000 with no issues.

Google set the bar and it’s cool to see the other model companies following suit.

Building AI apps for production uses cases has never been easier, and the trend doesn’t appear to be slowing down…