Hacker News Reader

Comment by sk11001 | Hacker News Reader

> This whole movement of simply increasing compute or throwing in more data to hopefully attain AI results is just a dead end.

There's no way for you or for anyone else to know that it's a dead end.

Also, it's not "simply throwing more compute" - the algorithms have evolved too and they'll continue to.

Also also, it's not "hopefully attain AI results" - we have AI results now and they keep getting better.

People are spending resources on a trajectory which has worked well - we've been cranking up the compute and better results keep coming out, why stop now?

View on HN

All I am saying is that there are other strategies to explore.

It's ok to have deep learning. But the herding towards deep learning is just too much.

Recall that before Hinton and his group presented thier record breaking ImageNet results, deep learning was a fringe technique that the likes of Marvin Minsky regarded as a failure in "Perceptions"

Scientific progress is made by researchers exploring other alternatives.

Herding can only slow down progress.

> All I am saying is that there are other strategies to explore

The “other strategies” have been completely sidelined. Even back when we had no transformers, only CNNs and RNNs, the other strategies were tossed aside because they didn’t show much promise in cifar or mnist or whatever benchmarks the NN community deemed the most suitable sota. Even before that, when we only had decision trees and catboost, people had already dismissed loess/splines etc. You can get a flavor of the dismissals here -

https://news.ycombinator.com/item?id=19145706

At this point that horse has left the barn.

The imagenet results’ “main” contribution was showing that scaling the compute on large datasets worked surprisingly well. They didn’t contribute any new models as much as they showed that scaling is what helped these models learn and obtain SOTA.

It’s no surprise that industry has been pushing for more compute ever since.

> All I am saying is that there are other strategies to explore.

Are people not exploring other strategies?

Do you know anyone in AI doing research on something that isn't Neural Networks?

They are very few. Too few.

> Herding can only slow down progress.

You don't know that either. What if there are actually no alternatives.

There are solutions. The human brain is one example.

Isn't that what we are trying to simulate?

No, and we should not

http://incompleteideas.net/IncIdeas/BitterLesson.html

One must consider the concept of "diminishing returns" eventually.

Has Generative AI Already Peaked? - Computerphile (https://www.youtube.com/watch?v=dDUC-LqVrPU)

Moores law is the inverse

Moores law doesn’t help when you’re spending money today rather than 10 years from now.

So much is being learned, and some stock prices have gone up. If nothing else, the people receiving coding suggestions and AI-generated artworks appreciate the billions of dollars and million programmer-hours spent.

Reply on news.ycombinator.com