Hacker News Reader

Comment by KKKKkkkk1 | Hacker News Reader

I don't know why this reminds you of Sutton's essay. Sutton claims that algorithmic work will be superseded by brute-force approaches powered by exponential growth in computing power. But this paper is not the result of exponential growth in computing power. It's the result of $1b worth of Azure credits provided for free by Microsoft.

Rich Sutton is a great scientist but he is fooled by randomness. He initiated his research program just as Moore's law was taking off. Thanks to Moore, his approach saw incredible success and brought him deserved acclaim. But just as Moore's law is pulling the rug from under him, he is using his stature to claim that no other approach but his can work.

View on HN

You have misunderstood Sutton's argument. Quoting Rich:

  One thing that should be learned from the bitter lesson is 
  the great power of general purpose methods, of methods that 
  continue to scale with increased computation even as the
  available computation becomes very great.

The point isn't that improvements in our algorithms is unnecessary or unhelpful, rather that the algorithms we should focus on will be capable of scaling with arbitrary amounts of compute/data. Such as, for example, neural networks, where we see an almost constant rate of improvement (for the appropriate architecture) as more resources are added.

I think this argument is neither here nor there. For computer vision problems, we use convnets, which are models inspired by a biological model of vision. By doing that we are embedding our preconceived notions of what vision is into our models instead of throwing compute and data at the problem. Earlier attempts using multi-layer perceptrons have been massive failures. Is this consistent with Sutton's analysis or contrary to it?

It's consistent with it.

Rich used to be very bullish on neural nets, then somewhat dismissive of them (due to the fragility/inadequacy of FCNs), and then increasingly enthusiastic as the renewed interest demonstrated that those problems could be overcome-- e.g., through better initialization, training, and (as you note) different architecture choices. His main concern was whether a method could keep working as more resources became available, as otherwise you would tautologically end up with something short of true artificial intelligence.

The important thing is that the technique can scale with increasing data or compute without hitting a hard or soft limit.

The engineering effort of utilizing 1B worth of Azure credits is surely worth at least 10% of it.

Yes then credit to azure engineers not open ai people

Wow, do you reasonably believe OpenAI people are not contributing?...

I thought HN visitors are not capable of that level of naive thinking.

Reply on news.ycombinator.com