Fast Convolutional Nets with Fbfft: A GPU Performance Evaluation
research.facebook.com
research.facebook.com
https://github.com/soumith/convnet-benchmarks
Check out the huge speedup between cuDNN(R2) and cuDNN(R3). Also, while there is some fantastic ML and DNN expertise at Facebook, there are only a few GPU experts there.
In contrast, NVIDIA is swarming with them. I would place my bets going forward on cuDNN. NVIDIA has bet the farm on deep learning at the expense of other problem domains. This is great for ML experts who want to add GPU support to their algorithms, and not so great for general HPC* (but then HPC just isn't as sexy as deep learning, now is it?).
*For example, cuBLAS perf has serious issues and cliffs on Maxwell GPUs. Sigh...
Unlike what the public perception is, NVIDIA, Facebook and other companies collaborate very very closely. NVIDIA's latest R3 release has FFT-based convolutions inspired by Facebook's work, and Facebook integrated some of the optimization suggestions for fbfft that NVIDIA gave. Further optimizations on the FFT approach (like tiled-FFTs) have been suggested and prototyped at FB and carried over by NVIDIA. The relationship is more of a deep collaborative effort.
At the end of the day, we want cuDNN to win, as it is a baseline system for all software.
However, if you want to do research, you want to have open-source code, because you often want to do funky changes to the base algorithm. NVIDIA falls short on this, and we at Facebook pledged to always keep our code open source (bitten by the fact that we had to start with closed-source blackboxes for our initial FFT work).
For Caffe is open source. And cuDNN makes it fast. And if it doesn't, just jump and down like the above report did, and it will be fast in the next release.
IMO if you're doing stuff for which cuDNN isn't good enough, you're probably capable of writing your own kernels, and that's awesome. Welcome to the Deep Learning equivalent of the 1%. Meanwhile, incorporate cuDNN into your production workflow to maximize comparative advantage.
Or don't, whatever...
I don't see how NVidia can really be displeased with this - this improvement continues to use CUDA and therefore lock users to NVidia hardware, failing to even acknowledge other solutions. People in this field are walking into a serious trap.
The OpenCL port Caffe has abysmal performance on AMD GPUs (see Caffe-CLGreenTea on Soumith's chart).
https://github.com/BVLC/caffe/pull/2195
Intel isn't even on the map yet, and there's no good training code for FPGAs. And ASICs? Maybe when Moore's Law slows down.
And it's not a trap at all. Now that DNNs are seen as a multibillion dollar industry, the status quo will change in a few years. But until then, it's NVIDIA's gravy train, and they earned it by taking a huge risk on GPGPU.
Now if you think you can build the competitor to NVIDIA and save us all from the looming trap, go for it! Meanwhile, all other things equal, use Caffe with cuDNN until someone earns your business with something better. As long as you're running at the API level, it really doesn't matter what's going on under the hood.