Edit: fixed links should be going live in a few mins, via: https://github.com/pytorch/pytorch.github.io/pull/141
808 karma · joined June 22, 2014
Edit: fixed links should be going live in a few mins, via: https://github.com/pytorch/pytorch.github.io/pull/141
The paper was authored by Sara Sabour, Nicholas Frosst, Geoffrey E Hinton in that order.
> Caffe2 is built to excel at mobile and at large scale deployments. While it is new in Caffe2 to support multi-GPU, bringing Torch and Caffe2 together with the same level of GPU support, Caffe2 is built to excel at utilizing both multiple GPUs on a single-host and multiple hosts with GPUs. PyTorch is great for research, experimentation and trying out exotic neural networks, while Caffe2 is headed towards supporting more industrial-strength applications with a heavy focus on mobile. This is not to say that PyTorch doesn’t do mobile or doesn’t scale or that you can’t use Caffe2 with some awesome new paradigm of neural network, we’re just highlighting some of the current characteristics and directions for these two projects. We plan to have plenty of interoperability and methods of converting back and forth so you can experience the best of both worlds.
https://discuss.pytorch.org/t/bayesian-computation-in-pytorc... https://discuss.pytorch.org/t/distribution-implementations/4...
With TF's XLA compiler, they are slowly getting towards kernel fusion, which will then reduce launch overheads.
We have similar things in the works for pytorch: to quickly JIT at runtime the dynamic graph that is getting executed. More news on this will come when time-appropriate.
The C libraries are shared among the Lua and Python variants
* Facebook * Twitter * NVIDIA * SalesForce * ParisTech * CMU * Digital Reasoning * INRIA * ENS
The maintainers work at Facebook AI Research
This repo is most definitely not ready for primetime.
Two comments:
1. I haven't seen an Intel team that does non-trivial DNN optimizations, almost all the (multiple) teams that have worked on optimizing neural networks have worked on trivial stuff like adding OpenMP optimizations to existing code, and integrating MKL into existing frameworks. Usually adding OpenMP optimizations isn't as relevant, because one uses CPUs for neuralnet inference, on a server where multiple processes are running. Per-core efficiency is much more important than cute threadpool optimizations like OpenMP which look great in micro-benchmarks.
The meaty and practically relevant optimizations have come from folks in the community like Andrew Lavin and Marat Dukhan ( https://github.com/Maratyszcza/NNPACK ) which are focused more on optimizing both single-core efficiency as well as multi-core.
2. The way this fork integrates MKLDNN into Torch is not great, where they add MKLDNN fields into the core Tensor / Storage struct, which is not great from a broader perspective. I will try to reach out to the authors and make this more proper.
p.s.: torch installs in a self-contained folder already as pointed by @gcr
Quotes from the paper:
Both char-CNN and VDCNN are trained on a NVIDIA Tesla K40 GPU, while our models are trained on a CPU using 20 threads.
Table2 shows that methods using convolutions are several orders of magnitude slower than fastText.
Our speed-up compared to CNN based methods increases with the size of the dataset, going up to atleast a 15, 000× speed-up.
Table 2 shows the speedups of:
ConvNets: 2 to 5 days on GPUs
FastText: 52 seconds on CPU
It's not perfect, but I cant give a reasonable answer to the other extreme of an opinion. fwiw, as a researcher I spent quite some time on this paper, but that subjective point doesn't mean anything to you.
The reigniting of deep learning around 2012 was because of Krizhevsky, Sutskever & Hinton winning the Imagenet challenge (1000 object classes)
Contrary to how much Google tried to sell Andrew Ng's "breakthrough" 2012 experiment with tons of PR, the paper is very weak, and cant be reproduced unless you do a healthy amount of hand-waving. For example, to get an unsupervised cat, you have to initialize your image close to a cat and do gradient descent wrt the input. Or else, you dont get a cat... It is not even considered a good paper, forget being breakthrough. Also, those 16000 CPU cores etc. can be reproduced with a few 2012-class GPUs and much smaller time-span than their training time.
The next slide after the 2012 breakthrough that shows the Javascript neural network -- contrary to what it looks like -- is not TensorFlow either. It has cleverly and conveniently been given TensorFlow branding, so most people just confuse it, but it's just a separate Javascript library akin to convnet.js
Since your page gets a ton of hits, it's at least worth it to publish a comment about these GLARING inaccuracies.
Karpathy's Arxiv-sanity helps a lot to keep in touch with the latest and greatest deep learning without having to spend all my time reading papers.
I work in deep learning and I think the article is more fear than food for thought.
Here are other reproductions just FYI:
Original Torch version from DeepMind: [github mirror] https://github.com/kuz/DeepMind-Atari-Deep-Q-Learner
Caffe: https://github.com/muupan/dqn-in-the-caffe