Hacker News Reader

Wouldn't it be better to express it like this: the authors haven't realized that the attention mechanism can be represented as a special case of Nadaraya–Watson kernel smoothing, a representation that hasn't resulted in practical improvements in attention mechanisms, probably indicating that the connection isn't that important in ML?

Like an accountant not realizing that addition and multiplication are special cases of a monoid operation.

View on HN

No. It definitely has resulted in practical improvements to attention. One practical improvement is that kernel smoothing is vastly easier to explain than "attention", which I think I have already demonstrated here.

Aside from that, kernel smoothing has roughly 70 years of literature devoted to every practical aspect you could imagine: approximation, accuracy, pedagogy, behavior under sparsity, relationship to other well-known theories like Bayeisan learning, and so on. Variants like Performer and elliptical attention, all directly arose from the knowledge that attention is intro-grad-stats kernel smoothing. A large portion of our understanding of ICL is directly or indirectly resultant from the fact of this correspondence.

Reply on news.ycombinator.com