I agree, I was really only proposing PRESS for the "parameter efficiency" part of your comment. It'd be interesting to see some modern takes on IRLS. I think generally this goes against the grain of the Cheap Gradient Principle which is why you see less of it (edit: eh, I think Fisher scoring can be cast in this light).

For instance, on modern modelling problems with non-linearities and change points, it's a lot less easy to do something like IRLS in an end-to-end system, but interesting as a research direction.