Sounds like we need some new training methods. If training could take place locally and asynchronously instead of globally through backpropagation, the amount of energy could probably be significantly reduced.
Yeah, I strongly agree. While Nvidia is working on better hardware (and they're doing a great job at it!), we believe that better training methods should be a big source of efficiency. We've released a new PyTorch library for efficient training at http://github.com/mosaicml/composer.
Our combinations of methods can train CV models ~4x faster to the same accuracy on CV tasks, and ~2x faster to the same perplexity/GLUE score on NLP tasks!
The degree to which energy and capital costs can be optimized will determine how large they can go.
We're at a point we we're turning into a computation driven society and computation is becoming a globally relevant power consumption aspect.
> global data centers likely consumed around 205 terawatt-hours (TWh) in 2018, or 1 percent of global electricity use
And that's just data centers, if you add all client devices you probably double that.
Plus that number will only continue to grow.
Why are people so dumb when it comes to planning for the future? Does it require a 1973 oil crisis to make people concerned about potential issues? Why can't people be preventative instead of reactive? Isn't the entire point of an engineer to optimize what they're building for the good of humanity?