> It’s logically obvious. You can recreate any linear regression using a neural network. So a neural network approach will always be at least as good as linear regression.
Well, maybe it's logically obvious, but it's also empirically false: https://www.youtube.com/watch?v=x7psGHgatGM (Skip to 13:36 to get to the point)
> But a neural network can model non-linear relationships as well. Now ask yourself, what is the likelihood of there being a non-linear relationship in the data? As with most real-world data, the likelihood of non-linear relationships is extremely high. That’s why we can be quite certain a neural network approach will able to outperform linear regression.
As I've said elsewhere in this thread, a deep neural network of N layers can be decomposed into a complex nonlinear function of N - 1 layers followed by a linear combination of the nonlinear features generated that function. There's no law that says that you have to use a neural network to generate your nonlinear features. You can use any method you like and then linearly combine those.