When I have worked on distributed systems, there are often several layers of caches that have nothing to do with latency: the point of the cache is to reduce load on your backend. Often, these caches are designed with the principle that they should not hurt any metric (ie a well-designed cache in a distributed system should not have worse latency than the backend). This, in turn, improves average latency and systemic throughput, and lets you serve more QPS for less money.
CPU caches are such a common thing to think about now that we have associated the word "cache" with latency improvements, since that is one of the most obvious benefits of CPU caches. It is not a required feature of caches in general, though.
The browser cache was built for a time when bandwidth was expensive, often paid per byte, the WAN had high latency, and disk was cheap (but slow). I don't know exactly when the browser cache was invented, but it was exempted from the DMCA in 1998. Today, bandwidth is cheap and internet latency is a lot lower than it used to be. From first principles, it makes sense that the browser cache, designed to save you bandwidth, does not help your website's latency.
Edit: In light of the changes in the characteristics of computers and the web, this article seems to mainly be an argument for disabling caching on high-bandwidth links on the browser side, rather than suggesting "performance optimizations" that might silently cost your customers on low-bandwidth links money.