We need to see something like "dropout" gain popularity again where you take a trained network and keep removing hidden neurons and re-training a few iterations until you can't obtain the same accuracy anymore, at which point you've reached what should be the smallest that network can be while still encoding the same knowledge. With models of these sizes though, few have the compute or the time to do stuff like that. There's a way of doing the whole thing iteratively though while you train that shouldn't add too much extra complexity..

The fact that the big players aren't doing this doesn't surprise me --- the more parameters, the less they have to worry about small-time competititors. I wouldn't be surprised if you could make an order of magnitude reduction in parameters and get the same results, and they're all just terrified we'll figure that out

As an undergrad I did some research on mapping "computational complexity" concepts onto neural networks to answer questions like what classes of problems require what minimum hidden layer sizes or network topologies using strategies similar to this. I was actually astounded how simple some of the resulting networks were that could nonetheless solve complex tasks like OCR with very few weights.