I thought the main selling point of deep learning that it finds non-linear connections in the data. Isn't it surprising that the implication is a linear separation of features?
There is lately a lot of hate against classic statistics on HN. I don't know why. Does it help to understand why and how NNs work? Not yet. But saying that it is utterly useless and won't provide any useful insights in the future sounds to me like telling the young Steve Jobs that dropping out of college and taking calligraphy classes instead of is utterly useless. And still, I am writing this on an Apple product, which set the standards for digital typography...
You didn't, but a lot of HNlers do. Maybe I should rant on them, that's true.
I'm thinking of some wonderful posts describing where/when/why linear regression can offer performance which is very close to the best from a NN-- except that regression models train much faster on much less data AND are interpretable.
Some discussion about how a NN works well for data where there is a lot of (statistical) structure to the data-- two close pixels in an image are likely to have very similar color/luminosity (and if not, the difference is important to the model, i.e. an 'edge'). But that NN don't do as well in a domain where the different features of the data don't have such relationships, say an econometric model or many biological models or ...
Agreed. Yet there are unknown feature spaces in which the deep nets separate the data linearly; the mapping learnt by the deep nets from the data to this space is likely highly nonlinear.
For example, a binary logistic regression can be thought as mapping data to a 1D space ([0, 1]), and separates the two classes at say 1/2 linearly.