Hacker News Reader

Comment by p1esk | Hacker News Reader

So it’s been almost two years since the capsules paper, and they still have nothing but mnist results? A bit underwhelming...

View on HN

Capsule networks are attempt to get past the limits of current approach to deep learning. It took 20 years until the the current paradigm in deep learning started to generate results. You start with toy problems and explore.

Hinton's "What is wrong with convolution neural nets?" is still true to the state of the art in convnets. Gradually turning features into objects like we do in deep learning is limiting. Distinguishing feature integration from object detection like humans do and how to do it is an open question. It can take another 10-15 years.

Or maybe it’s just another overcomplicated model that’s ultimately a dead end, like, say, RBMs, which were also pushed hard by Hinton (before DL breakthrough in 2012).

Has anything actually happened in the last two years? Genuinely curious.

I think there are two big things:

1. Attention mechanism (transformers). Really moved forward NLP. It allows long distance dependencies without the need for sequential processing (like in RNNs). Generally scales much better (see BERT/GPT-2/XL-Net/etc)

2. Separable convolutions. Significantly sped-up the standard CV networks. (mobile net, but now everything, really).

As a comparison, look at the transformer paper (also 2017), and the progress it has triggered.

https://arxiv.org/abs/1906.06818 says June 2019. To which paper do you refer?

"Matrix capsules with EM routing"

Reply on news.ycombinator.com