Hacker News Reader

Comment by irremediable | Hacker News Reader

I went to an interesting talk lately from an Nvidia research guy who believes GPUs are the "next" Moore's law. He's upbeat about this; he thinks it'll give a lot of jobs to lower-level programmers for the next decade or so.

View on HN

(1) GPUs are special-purpose. They're very good for doing the same floating point calculations, in parallel, on each datum - i.e. if your data can be represented as a picture. They are not much help for processing, for example, large amounts of text, or doing radically different computations on different data.

(2) In any sufficiently complex program, completely different computations are theoretically capable of running concurrently. At the very least, you need multiple CPUs/cores for that, and then you have the problem of assigning tasks to the different cores. Conventional languages are designed around a single processor and do not support this.

(3) Memory access is a bottleneck, which because of caching favours data structures which are compact over those which are scattered over a wide area of memory (such as lists, trees, etc.)

(4) CPUs were once designed to run specific programming languages, e.g. Burroughs mainframes ran Algol natively, and Lisp Machines ran Lisp natively. The PDP-7 and its successor the PDP-11 influenced the design of C. Nowadays, CPUs are designed without apparent reference to the languages which will be compiled onto them, but implicitly favouring the popular languages (C, C++, Java).

So what's happening, and what is likely to happen in future unless we do something about it, is that programs/languages are being shoehorned into processors, rather than different processors being designed to run different types of programs and languages.

Only if the GPU manufacturers start providing documentation on their chips.

I'll confess I've never done any CUDA stuff, so I don't know firsthand. Surely things are likely to get better, though?

AMD provides documentation for their GPUs and has for years. AMD pays a team to write free-software drivers to be included in the mainline linux kernel.

http://developer.amd.com/resources/documentation-articles/de...

Well, they used to. http://www.xbitlabs.com/news/other/display/20121107223140_AM...

Observe the @amd.com commits in this[0] driver supporting the latest AMD GPUs. Observe this[1] person who lists AMD as his current employer and describes his current job as "Working on open source drivers for Radeon GPUs."

AMD employs kernel developers who write free software drivers for their GPUs. That's all I was trying to communicate. I have no particular knowledge of the actual business organization.

[0] https://github.com/torvalds/linux/commits/master/drivers/gpu...

[1]https://www.linkedin.com/in/daenzer

Did he talk much about performance per watt? The trend with recent NV GPUs seems to be to increase performance but at the cost of ever-increasing power budgets; I'm starting to despair of ever getting anything that'll meet the Oculus reqs without sounding like a 747 taking off and/or melting a hole in the table.

Afraid not. He was talking to a load of machine learning researchers who want to use the chips for CUDA stuff, for whom power consumption isn't an issue.

Perf per watt has improved with every NV GPU generation and quite significantly. But it's true that the max TDP has increased too (but you get more in return).

GPUs are in an interesting place. They were not able to scale under 28nm due to the increased static leakage. Intel CPUs were able to scale because of dynamic power management, but GPUs cannot really use the same trick. It's only with the upcoming 16nm GPU generation that they got static leakage to a place where we see an actual new generation of GPU hardware. It's been about 5 years now of 28nm + GDDR4.

The lengths you mentioned are lambdas - factors for on-die features. This means that some part of the transistor cannot be smaller than lambda-size square. Usually, they are much bigger and often are of equal size even for wastly different lambdas.

Here's a table for SPARC CPUs, with process lambda, die size and transistor count: https://en.wikipedia.org/wiki/SPARC#SPARC_microprocessor_imp...

If transistors would reduce with lambda, then for two-fold reduction you would get four fold increase of density. Difference in density between T4 and T2 (90 and 40 hn processes) is less than two.

The static leakage you mentioned is mainly due to sizes of transistor parts.

You should also know that NVidia uses humans as chip designers (they can afford that), not an automatic process like some fabless companies use.

And now I get to my main point: NVidia overshot with 28nm and got transistors features that were too small, probably, due to not all that good software they used to calculate models. They continue to overshoot for 5 years until they corrected their overshoot. It has nothing to do with process lambda. It is process error in NVidia.

I just upgraded from a GTX 560 to a 960, and I've more than doubled my performance, and nearly halved the number of watts I draw, so it's not all bad.

Performance/watt is improving greatly! Look at this plot [1] from about 2013 showing the theoretical perf/w of CPUs up to Ivy Bridge, and GPUs up to NVIDIA Kepler GK110 and AMD Hawaii XT. Since than both Intel with Haswell (and Skylake on Desktop) as well as NVIDIA Maxwell and AMD Fiji have improved tremendously

The same article [3] shows clearly [2] that max TDP has increased a bit.

[1] https://www.karlrupp.net/wp-content/uploads/2013/06/gflops-p...

[2] https://www.karlrupp.net/wp-content/uploads/2013/06/tdp.png

[3] https://www.karlrupp.net/2013/06/cpu-gpu-and-mic-hardware-ch...

Reply on news.ycombinator.com