i've worked in this field for 6 years and have never heard of the 'shallow basin hypothesis', care to explain more? is it just the idea that there are many good solutions that can be reached in very different parts of parameter space?

all that grokking really means is that the 'correct', generalizable solution is often simpler than the overfit 'memorize all the datapoints' solution, so if you apply some sort of regularization to a model that you overfit, the regularization will make the memorized solution unstable and you will eventually tunnel over to the 'correct' solution

actual DNNs nowadays are usually not obviously overfit because they are trained on only one epoch