I have thought about that for a while and I wonder if it has to do with how memory becomes bigger more than it becomes faster. For example, compared to 30 years ago, PCs have about 1000x times more RAM, but it is only about 100x faster with about 10x less latency. It is a general trend for all sorts of devices and types of storage.
It means that for instance, storing an entire frame of video is nothing today, but in the analog times, it was hard, it means you simply didn't have enough storage for high latency. Now, you can comfortably save several frames of video, which is nice since more data means better compression, better error correction, etc... at the cost of more latency. Had memory be expensive and speed plentiful, a more direct pathway would have been cheaper, and latency naturally lower.