Hacker News Reader

Comment by astroalex | Hacker News Reader

I'm not an expert, but I became aware of this paper recently:

"A Battle of Network Structures: An Empirical Study of CNN, Transformer, and MLP" https://arxiv.org/pdf/2108.13002.pdf

I can't understand the technical jargon, but my interpretation is that MLP turns out to not be as good as CNN / Transformer. Maybe someone with more expertise can weigh in!

View on HN

MLPs discard more spatial information. That is great for "Do you see a cat?" but bad for "Where is the cat?". So which one is better depends on your use case.

MLP don't "discard" spatial information. Rather, spatial invariance is a very useful inductive bias for learning that MLPs don't have. If you had a perfect optimizer MLP would outperform CNNs (assuming hard enough tasks) but you don't have a perfect optimizer.

MLP is basically the vanilla neural network. The thing everyone sees first when getting taught about deep learning or similar. It turned out that (usually) MLPs are not very efficient to process matrix structures with it (e.g. 2d matrices like images). So, computer vision invented convolutional neural networks specifically to make image data highly efficient to process for neural networks. They outperformed MLPs in basically all aspects across the board. Transformers / Attention is a relatively new invention initially made to solve NLP problems more efficiently; but as it turns out, they work great on images as well.

TLDR - we've come full circle.

Reply on news.ycombinator.com