I'm curious how (absolutely) efficient the Transformer training is even on the TPUs. The results from self-attention models are really impressive but unfortunately their topology makes them very difficult to implement efficiently in silicon. It often becomes purely a question of memory bandwidth, because you're not doing much math per weight on each iteration. I wonder if the speedup is from the use of on-chip HBM in the TPUs.