CPU and GPU trends over time

Data parallel algorithms with high computation-to-communication ratios, good data locality and low local branch divergence will see these kinds of speedups when rewritten for GPUs. This is an important subset of algorithms, but don't expect GPUs to replace CPUs for more general-purpose computation any time soon.

It was interesting when Intel fail to deliver on their roadmap. Here is a link from 2000 where Intel was alleged to be saying 10Ghz by 2011 http://www.geek.com/articles/chips/intel-predicts-10ghz-chip...

Which of course lead to the great Multi-core debate (AMD started it, Intel followed with two physical die on a package later to integrate into a single die).

Which lead to the great Memory channel bottleneck (Intel was in-order single channel, AMD was multi-channel) and resolution (everyone now is multi-channel).

GPUs on the other hand have always had memory bandwidth issues so they went wide, the GPU equivalent of multi-channel, from the start. They also benefit from a feature of graphics which was 'do this same algorithm on all of these texels' which is straight forward to parallelize.

And as nuclear weapons designers, and genomic researchers will tell you, GPUs are also useful in doing MIMD simulations.

Its less clear what the parallel architecture breakthrough is for OLAP type problems (or parallel file systems as an even easier example) These have lots of operations in parallel but they also have high connectivity of data synchronicity across those parallel threads which, at the moment, brings everything down to earth.

Fun times!

1. What would be a first step towards this? It seems the big problem is that most code is sequential... and the opportunity is those parts that don't need to be, but just happen to be written that way.

2. Perhaps another opportunity is to try to remove the layers that make modern software so slow. Can they be distributed across all pipelines somehow?

3. Finally, a radical approach is not to try to convert existing apps, but to think about the whole problem of an app from the very beginning (what inputs will you get, what outputs do you want), and try to write that directly on a GPU. By making it concrete, you might see new tricks, perhaps by redefining the problem (the app's features).

And yet, all GPUs and CPUs of the whole world can just barely match one human brain (http://arstechnica.com/science/news/2011/02/adding-up-the-wo...). If that's not mindblowing, then what?