What are the characteristics of NVIDIA GPUs that make them superior for deep learning applications?
Phrased another way, if you're designing a card specifically to be good for training deep neural nets, how does it come out differently from cards designed to be good at other typical applications of GPGPU?
On the hardware end, Nvidia has slightly superior floating point performance (which is the only thing that matters for neural nets). Pascal also contains 16 bit floating point instructions, which will also be a big boost to neural net training performance.
I would be interested to hear about the difference in floating point performance. I would have guessed that, at this point, pretty much every chip designer in the world knows equally well how to make a floating-point multiplier. So it must be that nvidia vs amd have made different trade-offs in when it comes to caching memory near the FP registers or some such thing?