It's really a combination of memory latency and pipelining.
Memory latency is absolutely terrible compared to processor speed, and that has nothing to do with Moore's law. It's 60ns to access main memory, which is ballpark 150 cycles. If you have no caches, your 2.5Ghz processor is basically throttled to 16Mhz. You can buy some back with high memory bandwidth and a buffer (read many instructions at a time). But if you have no predictor, every taken branch flushes the buffer and costs an extra 150 cycles- in heavily branched code your performance approaches 8Mhz.
Then think about pipelining. We don't pipeline because Moore's law has ended. We pipeline because a two-stage pipeline is 200% as fast as an otherwise identical unpipleined chip. A sixteen-stage pipeline is 1600% as fast. Why the hell wouldn't you pipeline? Now, of course in the real world branched code can tank a deep pipeline. Which is where the branch predictor comes in, buying back performance.
http://stackoverflow.com/questions/4087280/approximate-cost-...