Now that I think about exposing cache hierarchy in a programming language, that makes much more sense. One can imagine a programming language with explicit control over CPU caches and their hierarchy. Also, this could make GPU/CUDA programming more explicit, safe and efficient.
Still, this requires the hardware to cooperate with software instead of pretending that random access memory is truly random access. This functionality just isn't there at this point.
Edit: this would require programmable caching hardware to make caching analysis and algorithms introspectable and changeable. For now, fixed caching algorithms implemented in hardware do provide lots of benefits.