Huang’s Law Is the New Moore’s Law, and Explains Why Nvidia Wants Arm
wsj.com
wsj.com
Which is cool I suppose but comparing it to Moore's Law seems a bit unfair.
And it's not even unreasonable. We collectively used transistors in general purpose CPUs in a way that wasn't most efficient but made it easy from a software and operational perspective. And that's changing now.
But multicore on a single (mostly) general purpose architecture (leaving aside vector instructions) is still easier from a software perspective than the very heterogeneous architecture--vector, array, FPGA, etc.--we're increasingly adopting.
are you referring to microsoft?
This was of course somewhat self-serving given all the criticism Intel caught for being slow to adopt multicore. But there were certainly concerns about software support at the time, especially on the desktop.
(In the end, it ended up not being as big a problem as some were fearing for various reasons. But it was a real fear at the time.)
Also it is extra silly because Moore's law is about transistor density and transistor cost.
The reason Moore's law was insightful was because he didn't make the simplistic generalization of computing power (which is more difficult to quantify anyway) and looked at what was really happening on a fundamental level from an engineering perspective.
Moore's law benefits all applications, by increasing performance across the board.
This new "Huang's law" increases performance for certain critical applications, while presumably decreasing performance of other applications (at fixed cost).
The difference is generally increasing transistor density vs. specializing hardware to perform critical tasks (like graphics rendering or tensor algebra).
Now the attention competition is at the article level, and the headline is near the top of the funnel. To not be subjected to (as much of) this insanity, you can use a smart aggregator like HN or a well-moderated subreddit.
Once again though did Nvidia need to buy Arm to do this? It could license its own IP and /or compete with SoC's of its own for this market. As in the server market the advantage of ownership (and what is worth $40bn) is the advantage it gives Nvidia in information and control of what is done with Arm IP.
I've been a couple times to one of the SF locations with coworkers. We definitely tried to trick system by passing items between us, and putting items down in random locations and then picking them back up later. When we left the store everyone got charged for the items they had on them.
I think you are too dismissive in saying its "just careful placement of the items and weight sensing".
Go has relatively few stores. Seems like the sort of thing you can still do manually, especially if the model expresses doubt.
At the scale they're operating at and the accuracy you point out, to me it's more plausible they're having humans assist.
I mean, I'm just asking the question how people know for sure Go isn't (at least partly) a Turk. It seems like an obvious thing to ask? But I first came to wonder if there's manual component involved after shopping a few times at Go. When it was busy, it took sometimes well over an hour to get a receipt. When it was empty, it was immediate. How come?
Also, the accuracy is completely anecdotal both ways... at least one person I visited the store with never got billed.
- the exact same article for those who don't have a wsj-account
Is this one of those PR articles [1] that try to convince public about the ARM-Nvidia deal being great for the industry? ( While I prefer ARM to stay independent, I am not exactly against Nvidia acquiring it )
After CUDA we had to use Tensor cores, later float16, now sparce matrix multiplications that work because deep learning uses lots of RELUs.
[1] https://youtu.be/fBRMaEAFLE0?t=651 "Free with every CD, a Cray supercomputer"
Money wise, the ARM acquisition for this amount of money makes no sense if they want to save on core, or architectural licenses.
With the amount of money, and engineers Nvidia has, they can easily bootstrap a CPU ISA on their own.
It was by far not their top priority.
And in their function relevant to the "AI" trend as giant linear algebra crunchers, GPUs don't need much help from anything else.
And in longterm, basically anything you do pales to logarithmic scaling of PPW/relative performance. A s..ttier, and cheaper chip few years down the line invariably beats current best.
In practical terms, the winner in the GPU/CPU/WhateverPU is decided by who gets the better fab contract.
1. Huang was seen sitting at the wheel of a Mercedes Benz S class, which has a stupendous amount of NVIDIA silicon. That's great, but the bulk of the market is for the electronics that go into a Chevy.
NVIDIA owns the highly profitable top end, but competitors (for gfx, robot brains, ...) are so disorganized that NVIDIA should move downmarket quickly to capture that terrain even if the profit margin is less.
2. A cynical take is that of "Patronage Networks". That is if Softbank can't manage "Return of Capital" to its investors, Masayoshi Son might come to his room and find a katana to cut himself with. The $40B from ARM goes a good way to "saving his ass" and might mean Huang owes him one in the same sense that politician owes something to voters.
If i could swap out a chiplet i might be inclined to agree with you.
tenth generation core mobile i7 (i7-10875H) 14nm, Q2-2020. 25mm^2 per set of two cores[0], 8 cores, 16 threads, 2.3-5.1ghz. 43 million transistors per square mm[1] -> over 4.3 billion transistors, 45W TDP.
So you have 4 times the threads, at a hair under twice the speed (without stressing turbo boost that much) at 1/3rd of the power at the wall. from top of the line desktop CPU in october of 2009 to "it's ok" mobile CPU from april of this year.
Intel stopped publishing (where i can find it) die size or transistor counts, i found the [0] and [1] information on a comparison between TSMC's 7nm fab runs and Intel's claimed 14nm fab runs; and i was only able to very briefly confirm this for the mobile i7 tenth gen and not the desktops. The desktop CPUs are like 400mm^2 larger in physical size, no idea about die size. Sorry. Intel stopped publishing those i guess, seems around the time that ryzen came out or slightly before.
edit: 11 years is 5.5 doublings of transistors IIRC, so you should really compare like to like, but i don't have that much free time. My laptop's CPU has 7.8 billion transistors just for the cores, and an additional 2 billion for I/O.
sorry. this information isn't in a precise place and i am unused to HN forms.
This is a developing area with a number of research and development efforts ongoing looking at basic packet filtering flows, for example: https://ieeexplore.ieee.org/document/7019193 https://arxiv.org/pdf/1312.4188.pdf
and AI-driven approaches like CyBERT showing promise: https://medium.com/rapids-ai/cybert-28b35a4c81c4
One of the main system challenges has been efficient high-bandwidth and low-latency delivery of packet flows between NICs and GPUs. NVIDIA has their cuVNF library (part of the Aerial SDK) that works with GPU Direct NICs and extends the DPDK toolkit to accomplish this: https://developer.nvidia.com/aerial-sdk
However, as for integrated devices, I'm familiar with at least one that claims to be on the market: http://www.h3c.com/en/About_Us/News___Events/News/201907/121...
For AI chip, will we fall into the same situation? will AI algorithm still focus on efficiency?
For example:
If you're solving NP problems (common in optimization), the size of the problem you can handle in a given amount of time is proportional to log(speed).
If you're adding features to software, then any given feature requires some amount of compute power. Experience shows these to be power-law distributed. Early software versions started with the cheapest features, so now we're adding things on the right side of the distribution. So each new feature is likely to require additional compute power increasing like a power law.
For a concrete example: when OSX added transparency and shadows to windows, it required every pixel in a window to be operated on twice. A doubling of a major component of the screen update time for a feature which only slightly increases user happiness/productivity. It had to wait until GPUs were fast enough to support it.
In machine learning, the accuracy of results increases much less than linearly, somewhere between log(# parameters) and sqrt(# parameters). And the inference time is linear in the number of parameters.
So it's a good thing that compute power is increasing exponentially or else progress would be really slow.
That said, speed as a derivative of how long the user waits has not changed simply because humans have not changed what they consider acceptable wait times.
PhotoStructure, a self-hosted photo and video manager, uses libvips (a wonderfully performant image library) and is multi-processed and multi-threaded. Typical images are >10x the resolution (24-40MP images are standard from DSLR) yet several are processed in parallel every second.
For networked applications, I feel like the much bigger limit is the network. The Tidal streaming service ran into this with its business model; streaming lossless, high quality audio is a bit of a moot point if the network doesn't allow you to download all of it without noticeable buffering when listening to music.
Seems pretty obvious when comparing, say, Duke Nukem 3D with Far Cry 5. Or Wolfenstein 3D with Assassin's Creed Odyssey.
We see it in video game rendering and weather prediction, but not in web page presentation, where it is squandered on pointless animation.
In Huangs law, si is commoditized, so the benefit accrues to the designer and software makers, and really only in narrow domains.
Nobody allowed Moore's Law to become a law.
The so-called Moore's Law is really just Gordon Moore's Extrapolation of an observation. In 1965, he observed that transistor count was doubling every ~12-18 months. He extrapolated that this trend could continue for ~10 more years to 1975.
But journalists and readers like snappy poetic words. "Extrapolation" is a long-winded 5-syllable noun so it's no surprise that the 1-syllable "law" was used instead. Wikipedia says: shortly after 1975, Caltech professor Carver Mead popularized the term "Moore's law".[1]
Although calling it "Moore's Extrapolation" would be more accurate, it just doesn't roll off the tongue as nicely. Sometimes humans value the sing-song poetry of a meme more than its dictionary accuracy.
[1] https://en.wikipedia.org/wiki/Moore%27s_law#History_of_the_c...
If it's about device physics or transistor and single-gate layout, it is likely Mead. If it's about organizing logic functions on a chip, it is more likely Conway.
She relied on Mead for a lot of the publicizing of her design methodology, because he was better on stage.
The business application of it meant profits from each generation could fund the higher costs of each succeeding generation.
Now we are in the tens of $billions for a new generation, with much smaller increases in performance, and it is faltering. Will we spend hundreds of $billions on factories for smaller transistors? Or will we find a more cost-effective way to get more performance?
But the expectation is that a lot of continued performance gains will have to come through other means. These are some of the things that were talked about at the Hot Chips conference: https://enterprisersproject.com/article/2020/9/moores-law-wh...