Here’s the challenge: could you ever specify an objective function concretely enough to sustain an assault by aggressive optimization? https://thezvi.wordpress.com/2019/12/31/does-big-business-ha...

In all our mathematical experience, the answer turns out to be that we can only ever define an approximate objective function, and we almost always need some kind of “regularization” to constrain our optimization strategy. And in many cases, an effective regularization practically (which results in better “generalization” technically) might be “early stopping” I.e. not optimizing as hard!