The real challenge is probably in the power efficiency. As far as I know right now power consumption is the biggest cost factor when training new models. Everything else seems like a minor trade-off but when accounting for the power consumption there is real money involved.