I don't understand how papers like this end up on the homepage of HN. The abstract makes very broad claims, but the paper itself only runs toy experiments on tiny datasets with tiny neural networks.

No good ML practitioner believes that tiny, shallow, fully-connected neural networks are the best algorithm for every problem (just look at Kaggle results if you don't believe me). Especially for small, easy datasets like the ones analyzed in this paper, NNs are often not the best choice. However, for large scale image classification, density modeling, sequence modeling, etc. (none of which are tested in the paper), NNs are SOTA.

Hilariously, the paper only compares polynomial regression to tiny neural networks. I bet if they had thrown in results from XGBoost or other classical ML techniques, polynomial regression would be blown out of the water.

There's also some heuristics you can use to tell when a paper like this isn't necessarily representative of the field of ML:

- Missing citations (e.g. "It is well-known that NNs are prone to overfitting [Chollet and Allaire(2018)], which has been the subject of much study, e.g. [?].").

- Inconsistent formatting (e.g. the tables on page 7)

- Sentences like "Much more empirical work is needed to explore these issues." following something that sounds like an easy experiment to try.