Stacked Capsule Autoencoders | Hacker News Reader

Stacked Capsule Autoencoders

akosiorek.github.io

46 points_Microft9 comments

So it’s been almost two years since the capsules paper, and they still have nothing but mnist results? A bit underwhelming...

Has anything actually happened in the last two years? Genuinely curious.

As a comparison, look at the transformer paper (also 2017), and the progress it has triggered.

I think there are two big things:

1. Attention mechanism (transformers). Really moved forward NLP. It allows long distance dependencies without the need for sequential processing (like in RNNs). Generally scales much better (see BERT/GPT-2/XL-Net/etc)

2. Separable convolutions. Significantly sped-up the standard CV networks. (mobile net, but now everything, really).

https://arxiv.org/abs/1906.06818 says June 2019. To which paper do you refer?

"Matrix capsules with EM routing"

Capsule networks are attempt to get past the limits of current approach to deep learning. It took 20 years until the the current paradigm in deep learning started to generate results. You start with toy problems and explore.

Hinton's "What is wrong with convolution neural nets?" is still true to the state of the art in convnets. Gradually turning features into objects like we do in deep learning is limiting. Distinguishing feature integration from object detection like humans do and how to do it is an open question. It can take another 10-15 years.

Or maybe it’s just another overcomplicated model that’s ultimately a dead end, like, say, RBMs, which were also pushed hard by Hinton (before DL breakthrough in 2012).

Why do people assume that the assumptions in DL are correct for more general intelligence? It's not remotely like a real biological NN. It's seems like an arbitrary gross approximation that everyone is starting with just because it is the prevailing worldview.

Why do some Chinese focus on SNNs? I suspect they just have a different worldview because they are exposed to a different information stream in their language. But SNNs are still a gross approximation. Also, why assume it needs to look like an NN at all?

Reply on news.ycombinator.com