Google’s Cloud TPU Pods are now publicly available in beta

The underlying principle is that even if the silicon isn't strictly faster, you can shove a dumb amount of data through with the right architecture. And efficient training with data parallelism means that it's a worthwhile strategy.

I can, actually. :) Adding 10k tensor cores to a GPU would not make it run much faster, and would be prohibitive in terms of die space. Moreover getting rid of the 5,000 FP32 cores would slow down DL workloads significantly.

The 640 Tensor Cores vs 5,000 F32 cores comparison is misleading, because they are not measuring the same thing.

An "FP32/64 core" corresponds to a functional unit on the GPU streaming multiprocessor (SM) which is capable of doing one scalar FP32 operation. One FLOP, or maybe two if you are doing an FMA. V100 has 5120 FP32 units and 2560 FP64 units.

In contrast, a "Tensor Core" corresponds to a functional unit on the SM which is capable of doing 64 FMAs per clock. That is, a Tensor Core does 64 times as much work as an FP32 core. Integrated circuits aren't magic, if you're doing 64 times as much work, you need more die space.

Moreover, there is nothing to say that nvidia isn't able to use some of the same circuits for both the fp32 and tensor core operations. If they are able to do this (I expect they are) then reducing one does not necessarily make space for the other.

Increasing the number of tensor core flops by a factor of 20 (500 -> 10,000) would not make the GPU 20x faster, probably not even 2x faster. This is because you will quickly run into GPU memory bandwidth limitations. This is not a simple problem to solve, nvidia GPUs are already pushing what is possible with HBM.

Lastly, although you're correct that, in terms of number of flops, most DL computation is done by tensor cores (if you've written your application in fp16), that doesn't mean we could get rid of the f32 compute units, or even that significantly reducing their number would have minimal effect on our models. Recall Amdahl's law. We usually think about it in terms of speedups, but it applies equally well in terms of slowdowns. If even 10% of our time is spent doing f32 compute, and we make it 10x slower...well, you can do the math. https://en.wikipedia.org/wiki/Amdahl%27s_law

Indeed, I was just looking at an fp16 tensor-core cudnn kernel yesterday, and even it did a significant amount of fp32 compute.

The implicit argument I read in parent post is that nvidia could build a significantly better DL chip "simply" by changing the quantities of different functional units on the GPU. This is predicated on nvidia being quite bad at their core competency of designing hardware, despite their being the market leader in DL hardware. It's kind of staggering to me how quickly nonexperts jump to this conclusion.

Here's a talk I gave at cppcon about much of this (note that it's pre Volta). https://www.youtube.com/watch?v=KHa-OSrZPGo&t=1s

As to deployment, once you figure out what works, there's also libtorch, or if that doesn't work for you for whatever reason, implementing a model you know working hyperparameters for can be done in a day or two on whatever framework works for your backend, including TF.