Hacker News Reader

Comment by mrfusion | Hacker News Reader

Is a high end computer anywhere near 20 quadrillion calculations per second?

Edit: 20 quintillion would be 20 million teraflops. Sound right?

If you interpret "calculation per second" as a FLOPS [as measured on LINPACK], 20 quadrillion is 20 PFLOPS. 20 PFLOPS puts you at #7 on TOP500.

A looser interpretation is that you could count the number of integer adds you can push through a second multiplied by vector width. The CPU component of this would be in the region of 1.5 GHz (AVX-512 clock-speed) * 64 (8-bit adds/512-bit vector) * 2 issue ports * 10-20 cores (I don't know where the $4,000 in 1999 dollars hits in terms of pricepoint here [1]). You're looking at the region of low single-digit TOPS here. I don't know enough about the physical architecture of GPUs to estimate their theoretical headcount here, but I'd be surprised if you could push much past the teens of TOPS on those machines either.

So, for the price point he predicted, we're off by a factor of 1,000.

[1] I'm counting physical cores here. Hyperthreads don't add more actual ALUs, they just let two independent threads fill these up with operations.

If I didn't make any mistakes 20 quadrillions are equivalent to 20000 TFLOPS. Wikipedia says you can get 12 TFLOPS with a high end graphics card for 1200$. 4000$ inflation adjusted is 6000$ which means you can get 5 of those for a total of 60TFLOPS which means he's off by a factor of 300. If you used CPUs instead (which is probably what he meant at the time) you can get 225 GOPS (integer operations, not floating point) for 374$ with a Core i7-9700K (source: wikipedia). With 6000$ you buy 16 of those for a total of 3.6 amazing TOPS, which makes Ray off by a factor of 5000.

I think it would be fair to allow tensor-ops instead of generic flops, though, and in that case he would "only" be behind by about 1-2 orders of magnitude.

RK's estimates are consistently best case estimates. But even if his estimates are wrong by a couple of decades, a lot can still happen before 2050.

The question of how to count operations is tricky, since a lot of it depends on what the actual problem you're trying to solve is. LINPACK is the favored benchmark for supercomputers, but that is more out of consistency with the past than for actual reliability of its metrics.

What the prediction was trying to gauge is how much a computer that is equivalent in computational power to a human brain would cost. I think a case could be made that the human brain's computational power is not equaled by the faster supercomputer today, at which point he was off by several orders of magnitude (in terms of price).

As you say, it is tricky. Still, the current trend is that TPU's are taking over for CUDA-like GPU's for solving AI problems (such as with Alpha0), so it could be argued that tensor operations is the most relevant metric for estimating when human level AI will be possible.

Of course, tensor operations do not map 1:1 vs FLOPS, even for AI problems. On the other hand, tensor oriented hardware is currently getting faster at a higher rate than traditional GPU's.

But RK's prediction did assume that we would have reached full 3d chip manufacturing processes by now, and we may not be able to have human brain levels of performance on a single die until we do, which looks like it will take 5-10 years.

To be clear about one thing:

When LINPACK measures FLOPS, it doesn't count how many instructions the computer executed. It uses a formula for how many floating-point operations it takes to solve an NxN linear equation and combines that with how long it took the computer to execute. So NVidia's tensor instructions actually contribute tens of operations per instruction retired.

The prediction is attempting to model when a computer will exceed human computational power, which was estimated by Ray Kurzweil at 20PFLOPS, although it's unclear to me if we was specifically thinking of LINPACK Rmax or some other metric.

The last time I did the calculation we’re still lucky to get over a Tflop/s on a modern processor (if I did my math right). The 5 ghz wall and core counts contribute to the stagnation. The real hardware progress now is in non86 processors enabling higher core counts for the $, memory speed, and the eventual reality of permanent storage to be as fast as ram. Ie we’re likely to get “there” by increasing core counts and inter-process communication speeds.

Only supercomputers are in the range. https://en.wikipedia.org/wiki/TOP500#Top_500_ranking Number 10 in the list for example can do 17 quadrillion calculations. (10^15 = petaFLOP (https://en.wikipedia.org/wiki/FLOPS).

This seems to match my napkin calculation of this prediction being off by around a factor of 10^5 (~100.000 processors being needed to match it). Although in the grand scheme of these predictions that's probably not too way off what it probably shows well is that the exponential curve Moore's law had given us is definitely not holding anymore. Since Kurzweil had used it or something similar to extrapolate his numbers this seems to bring into question the whole premise of the singularity prediction.

Not that I'm much into "predictions", but for $4000 you can get a lot of GPU cards (assuming you can squeeze them all into a box, with the necessary power and connections).

I was 1999 dollars so maybe 5 or 6k today?

Do all the gpu cards add up to 20 quintillion?

Typical $500 graphics card today will do around 10 teraflops. You'd need 2k of them ($1m) for 20 quadrillion flops (the prediction is quadrillion not quintillion).

You can't really get a lot of GPU cards for $4000. Even two-year-old mid-range cards are $500/ea. The current high-end cards are $1500/ea.

Reply on news.ycombinator.com