zak

That's a great question. I don't have that analysis handy, but it would definitely be worth doing.

Thanks for the feedback. As more and more new ML system architectures emerge over the next few years, these performance comparisons will get even more complicated. Chip-to-chip comparisons and server-to-server comparisons are arbitrary, and it will sometimes be difficult to define these boundaries at all.

Our impression is that top-line performance comparisons independent of system size (like the comparison in the blog post) and performance-per-dollar comparisons are the easiest to understand, but we're certainly open to other ideas.

Author of the blog post here.

We submitted multiple results using various Cloud TPU v3 Pod slice sizes to show the current achievable Transformer training efficiency at several scales:

https://mlperf.org/training-results-0-6

We're actively improving the whole TPU software stack, so training efficiency is likely to continue to increase over time.

Author of the blog post here.

Cloud TPUs are designed to maximize performance-per-dollar, so you are right that pure performance comparisons at maximum scale don't tell the whole story.

The most straightforward performance-per-dollar comparisons would be among several different hardware configurations across the major public clouds. However, we haven't yet seen any other public cloud MLPerf submissions at scales comparable to Cloud TPU Pods, so there isn't currently a strong baseline available for comparison. It's also not clear whether public cloud networking will ultimately be able to match the performance of the network hardware that was used to produce the largest-scale on-premise MLPerf submissions.

Actually, anyone can use a Cloud TPU v2 (one of the accelerators mentioned in the blog post) for free via Colab: https://colab.research.google.com/notebooks/tpu.ipynb

Several other TPU-enabled Colabs are linked from the TFRC website: https://www.tensorflow.org/tfrc/

At present, preemptible Cloud TPU v2 and v3 devices are widely available, and they are likely to be the most cost-effective option for training any of the models listed here, often by a wide margin: https://cloud.google.com/tpu/docs/tutorials

Definitely appreciate your point about comparing Summit to your desktop - however, the difference is that you can't rent Summit via any public cloud, whereas you _can_ rent Cloud TPU Pods.

You might find the comparison between 8 x V100 GPUs on GCP and a full Cloud TPU Pod more relevant - in that case, as of the time the Google Cloud blog post linked above was published, a full Cloud TPU Pod delivered a 27X speedup at 38% lower cost for a large-scale ResNet-50 training run, all without requiring any code changes to scale beyond a single device.

+1 on open sourcing the code. If you post it somewhere, we'll take a look to see why the Adam optimizer isn't behaving as expected in your implementation.

It may also be helpful to compare your code with the Fashion MNIST Colab example here: https://colab.research.google.com/github/tensorflow/tpu/blob...

Here is a larger-scale comparison of Cloud TPU and Google Cloud GPU performance and cost (focused on Cloud TPU Pods): https://cloud.google.com/blog/products/ai-machine-learning/n...

All the code used in that comparison is open source, and there is a detailed methodology page with instructions that you can follow if you want to reproduce the results: https://github.com/tensorflow/tpu/blob/master/benchmarks/Res...

Also, Cloud TPUs are available to everyone for free via Colab. Here is a sample Colab that shows how to train a Keras model on the Fashion MNIST dataset using the Adam optimizer: https://colab.research.google.com/github/tensorflow/tpu/blob...

(I work on Cloud TPUs)

We're on it! = D

You can also use Cloud TPUs for free in your browser via Colab. Several sample notebooks are available here: https://www.tensorflow.org/tfrc/