One of the main things I've noticed is that several of the techniques are well intentioned for other reasons. For example, when we debug low cache rates with users probably culprit #1 is usually some memory system. Many of those memory systems are trading a stable prefix for the ability to augment the current turns request with some new memory data. For some -- this may be the right trade-off, but the impact is slower responses times due to the extra prefill compute.

So when a memory system promises better LLM performance the other trade-offs aren't necessarily well understood (at the expense of caching).

Features like OMP trying to clean up history to shrink context I think are also well intentioned. But I found these trade-offs interesting hence wanting to share.

It's easy to break the cache and there's a number of very valid usecases (mostly context management such as compaction or tool shaking).

Not to mention that it can be very strategic to break context. E.g, after chasing a bug for a while it does make sense to summarise all the approaches that did not work so that you have more context available for newer approaches.

Getting 99% cache hit rates while providing a good user experience is the hard part!