Hacker News Reader

Comment by _bxg1 | Hacker News Reader

It could be that the slowing of Moore's Law results in a more diverse array of specialized computing hardware. After all, the resurgence of ML didn't happen because x86 got fast enough, but because video games - of all things - funded the maturation of a whole new category of massively-parallel chips.

View on HN

Moore's law never slowed. At worst, one can rightly say Intel had issues with 10nm.

https://docs.google.com/spreadsheets/d/1NNOqbJfcISFyMd0EsSrh...

the resurgence of ML ... because video games - of all things - funded the maturation of a whole new category of massively-parallel chips

Fake news. It has far more to do with the rise of distributed computing than the existence of GPUs.

I don’t really see this; All of the currently popular machine learning frameworks either support GPU or are oriented around GPU based execution. For most developers working on AI it is still the defacto standard.

I think.. both. Distributed computing (Hadoop) was why a lot of data got collected and made available(ish).

GPU's are the engines that made CNN's (in particular) tractable, and opened up a bunch of applications for many companies, and opened up a reasonable route to results for a generation of researchers.

Can anybody elaborate on why this is downvoted? This would be my guess as well, simd parallelism of GPUs solves only part of the challenges, you still need a general purpose data crunching machine to prepare and handle learning data.

For one GPU speedup over CPU isn’t that dramatic for small to medium sized problems, e.g. MNIST or CIFAR that one would try algorithm ideas on. So I think it’s a stretch to see GPU as essential to the new algorithms. On the other hand for large problems like the original Alpha Go you need to figure out the distributed computing to really scale.

This isn’t to say that GPUs aren’t nice. They do save time or for the same amount of time let you produce more polished results, which means in a competitive environment everyone would use them.

Exactly. GPUs are necessary now, but did not originally herald the deep learning revolution.

Except that's "currently". NNs can and were trained on CPUs originally.

Don't call a statement you disagree with "fake news". It a) doesn't even make sense in this case, and b) distracts from the discussion.

I don't think this is very true. You can trace the deep learning revolution back to VGG and a fundamental driver in the success of the first multi-level networks was the ability to train in semi-reasonable amounts of time using GPUs.

Even today distributed training is relatively uncommon while pretty much everyone uses NVIDIA GPUs.

VGG was a long time after the first deep learning revolution.

You could say it started with ImageNet 2012 where AlexNet showed that deep networks were a newly promising area of study - however the actual performance of AlexNet was very very far from human performance. I tend to say the revolution started at ImageNet 2014 with VGG/GoogLeNet, the first human-caliber performances. Or you could say it was ImageNet 2015, when the first ResNet had better-than-human performance.

Reply on news.ycombinator.com