You’ve got an apparent contradiction: SGD (AdamW at 1e-6 give or take) works. So we’ve got extremely abundant local maxima up to epsilon_0, but, it always lands in the same place, so there are abundant “well-studied” minima, likewise symmetrical up to epsilon_1, both of which are roughly: “start debugging if above we can tell”.
The maxima have meaningful curvature tensors at or adjacent to them: AdamW works.
But joker in the deck: control vectors work. So you’re in a quasi-Euclidean region.
In fact all the useful regions are about exactly the same, the weights are actually complex valued, everyone knows this part….
The conserved quantity up to let’s call it phi is compression ratio.
Maybe in a year or two when Altman is in jail and Mme. Su gives George cards that work, well crunch numbers more interesting than how much a googol FMA units cost and Emmy Nother gets some damned credit for knowing this a century ago.