> they do not quite care about parameter efficiency.
Google Research is pretty big, I used to think like you did but I think it's mostly b/c DeepMind just hogs all the spotlight.
Check out PRESS [0] for example.
Google Research is pretty big, I used to think like you did but I think it's mostly b/c DeepMind just hogs all the spotlight.
Check out PRESS [0] for example.
But you gave me another point to support my view: PRESS uses stochastic gradient, not second-order method like IRLS.
For instance, on modern modelling problems with non-linearities and change points, it's a lot less easy to do something like IRLS in an end-to-end system, but interesting as a research direction.
The very SGD thing was developed because it was the only way to train something like neural network with small memory and, more importantly, in reasonable time. Multiplying of training time by N (number of parameter) meant having good result in a year, not in a day.
And today we have large batch training with complex synchronization systems to speed up training even more. Which bring us closer to the whole-dataset training and, I guess, second-order optimization as well.