> it does so using a network with 480 million parameters. The training to ascertain the values of such a large number of parameters is even more remarkable because it was done with only 1.2 million labeled images—which may understandably confuse those of us who remember from high school algebra that we are supposed to have more equations than unknowns.

This is one of the key misunderstandings are still deeply rooted in people's minds. For modern DL, a large part of the learning comes from "internal" data points, in this case the pixels of the image, as opposed to the labels. If you count the number of pixels, you will likely get something like 1.2 trillion, more than enough to justify the 4.8e8 parameters. It's the usage of internal data that prevents overfitting, NOT the random initialization and SGD as claimed in the article.

Another way to see this is: if you need more labels than parameters, how can GPT3 have ANY parameters at all? It is trained purely on raw text data.