Google is certainly Nvidia's top customer.
One might even conclude that one person inside Google had written a brawny and influential paper that was right in a lot of ways, but wimpily dead-wrong to apply to GPUs.
Now we all must own our choices and one was very silly, stupid, and naive to blindly accept a position at Google, but everyone makes mistakes, no?
Further, TensorFlow looks fantastic but given that even just a year ago, when one was recruited to return to Google for !(Deep Learning) but for something which also happened to be another braindead obvious choice to run on GPUs, one was informed said viewpoint on GPUs was still in effect and declined.
So what I'd really love to know is how the message finally sunk in? I'm betting there's a really great story here.
That sucks, but that kind of thing happens a lot. It's not that surprising. Google wanted a homogeneous infrastructure for a long time. But a new application (neural networks) motivated a new infrastructure (GPUs in clusters).
There was that Stanford paper everyone talks about which compared training a neural network on Google's 16K cores (DistBelief paper) vs a handful of GPUs with infiniband. Even Andrew Ng has seemed to subtly criticize Google for thinking "the old way" (i.e. "cloud" technology vs HPC technology, HPC being more effective for neural nets)
Also, I don't quite understand your comment about brawny vs. wimpy. Yes Urs wrote a rebuttal against wimpy cores. As far as I remember, his argument is basically that there is some portion of software on the critical latency path that's not multithreaded. wimpy cores make the most sense in a world of perfect parallelism.
And from a professional standpoint, I moved on and fixed the damage I let them inflict on my career. I eventually built exactly what I was told couldn't and shouldn't be done with GPUs.
But back then, the argument Google made to me was that too much of the workloads I wanted to run on GPUs were serialized (hence brawny versus wimpy). And from a classical parallel algorithms perspective, they were correct. The problem is that from a GPU programming perspective, they weren't even wrong(tm). And at the time, I had ~5 years of CUDA programming (very early adopter) and 30,000+ lines of CUDA code to back that up whereas the people I tried to convince had zero such experience so there was just no way. My argument boiled down to TLDR: O(n^2) or higher probably belongs on GPUs. Would anyone disagree these days?