Hacker News Reader

You mean this speedup?

> 1024 TPUs are twice as fast as 480 GPUs

Might it be because there are twice as many?

Author of the blog post here.

We submitted multiple results using various Cloud TPU v3 Pod slice sizes to show the current achievable Transformer training efficiency at several scales:

https://mlperf.org/training-results-0-6

We're actively improving the whole TPU software stack, so training efficiency is likely to continue to increase over time.

What I'm really asking is: how much effective compute throughput are you able to get during Transformer training relative to the amount of theoretical raw compute available?

That's a great question. I don't have that analysis handy, but it would definitely be worth doing.

What about energy? (watt-hours?)

Reply on news.ycombinator.com