Hacker News Reader

My intuition for this is that CNNs impose a stronger prior on the function space. We rely on the previous experience of computer vision and even our own retinas to give it a useful prior.

Transformers are more general, they impose a weaker prior and require more data. But by imposing a weaker prior they also are more flexible and adaptable, thus able to take advantage of more data.

View on HN

Not all vision transformers have weak priors. Shifted-window transformers and neighborhood attention have priors well suited to images; the latter is showing extremely strong performance in image generation (such as the recent hourglass diffusion which allows pixel space diffusion to be trained orders of magnitudes faster than vanilla attention) and general image classification, and certainly does not need the same dataset size of classical ViTs.

Agree. However I feel that there may still be many priors that are useful in practice, like decomposing the vision task into a hierarchy of sub problems, but as a human - I could be biased.

Oh absolutely, I think having a prior makes these problems tractable. Otherwise we would just force feed data to fully connected networks!

It would be much better if we had a way to learn good priors -> extract them -> initialize new models with them. We shouldn’t be trying to guess or handcraft them.

Reply on news.ycombinator.com