A trivial example is BERT_large; won't fit into 12/16GB and takes ~year to train from the scratch on a single 15TFlops machine. It's now a base model for transfer learning for NLP.
Also you are talking about a model which is specifically designed for TPU (the dimensionality of the networks is especially fine-tuned).
And even tho, BERT_large still fit in the memory of a single GPU (for very small batch), there is an implementation on Pytorch. I don't understand, are people complaining that Deep Learning is actually (reasonably) scaling ? Isn't it a good news ?
MS already published another model based on BERT that is even better. It's unlikely memory x #GPUs would go down in foreseeable future; it's more like that everybody will start as large models as their infrastructure allows if they find something that improves target metrics.