It's no surprise that in domains where low-parameter models work at all, they often work better than NNs. In econometric forecasting, where the traditional methods encapsulate more prior knowledge than could ever be inferred from the data, neural networks aren't anywhere near competitive. Even in domains where NNs are state-of-the-art, they might not be worth the training costs over GLMs or gradient-boosted forests or whatever.
But there are many other domains, like raw waveform modeling, large natural images (not MNIST), or NLP, where anything that isn't neural fails miserably. These are the domains where people are most excited about neural networks — problems that otherwise have no solution at all. Neural networks won't help you get a better interpretation of your n=25 daily-resolution cohort study, or predict user retention from your 300-column, 20K-row Excel spreadsheet.
In their paper, the authors dismiss most of NN's greatest recent successes as being "specialized" applications that only work because they incorporate tricks like recurrence and convolution, and not because they are neural. This is a much clearer, bolder claim, but they hardly support it as far as I can tell. The things that they call "specialized" (RNNs and CNNs!) are what most would recognize as the bread-and-butter of neural network design.