It's ok to have deep learning. But the herding towards deep learning is just too much.
Recall that before Hinton and his group presented thier record breaking ImageNet results, deep learning was a fringe technique that the likes of Marvin Minsky regarded as a failure in "Perceptions"
Scientific progress is made by researchers exploring other alternatives.
Herding can only slow down progress.
The “other strategies” have been completely sidelined. Even back when we had no transformers, only CNNs and RNNs, the other strategies were tossed aside because they didn’t show much promise in cifar or mnist or whatever benchmarks the NN community deemed the most suitable sota. Even before that, when we only had decision trees and catboost, people had already dismissed loess/splines etc. You can get a flavor of the dismissals here -
https://news.ycombinator.com/item?id=19145706
At this point that horse has left the barn.
It’s no surprise that industry has been pushing for more compute ever since.
Are people not exploring other strategies?
They are very few. Too few.
You don't know that either. What if there are actually no alternatives.
Isn't that what we are trying to simulate?