Seemed to take terrible performing models and marginally improve them. None of them are fit for purpose at the end of the training, so what was the point?
This experimental process on small NNs us exactly how science is done. You develop and test theories of what will work using small networks because you can do them quickly.
Then you test on large, production size networks based on the lessons from the small ones.