Attention is not based on Euclidean closeness, therefore it is not kernel smoothing. Saying otherwise would be the same as saying kernel smoothing is just an average (e.g. omitting the important part that it has to be distance-based-weighted).
First and most importantly, attention is, formally, kernel smoothing with a particular exponential kernel function, and if you don't believe me, you can read the paper I linked, which proves it.
Second, and less importantly: kernel smoothing absolutely, definitely does NOT require a kernel function representing Euclidean distance, and in fact the entire point of kernel smoothing is that you can pick basically any function that fits your needs. Many use cases find it helpful to have a real-valued kernel function that is positive and semidefinite, or nonnegative and integrating to 1, or whatever. Different kernel functions in these spaces have different properties and trade-offs. The exponential kernel attention uses (if memory serves) is (1) finite and nonnegative, and (2) has a positive total weight so you can normalize.
Third, and least importantly, you definitely can have a kernel function that is "uniform" (i.e., "just" an average) although it's generally not very helpful (e.g., not positive semidefinite).
Well, here's the thing. Kernel smoothing uses Euclidean distances. "with a particular exponential kernel function" is the novel bit. I even gave an example of what absurdity you will get if you allow arbitrary addition like this to the term.
> every assertion here is mistaken. ... Second, ..
Never made that assertion.
> Many use cases find it helpful to have a real-valued kernel function that is positive and semidefinite, or nonnegative and integrating to 1, or whatever
Yeah, that's the kernel. https://en.wikipedia.org/wiki/Kernel_(statistics). What you feed to the kernel is Euclidean distance. That's where the difference is.
> you definitely can have a kernel function that is "uniform"
You did not get that point. See response to your "First". I was explaining why the reasoning that led you to "attention == kernel smoothing" is faulty - if you apply it to averaging, you will get "averaging == kernel smoothing". Because kernel smoothing is "just" averaging with distance based weights. So attention is "just" kernel smoothing with Q-K based distance function. The issue is in both cases the part after "with " does all the heavy lifting.
For those following along, when this user says that attention is kernel smoothing with a "Q-K based distance function", I think (THINK!) what they're trying to say is the second part does more work than it gets credit for when we say attention is "just" a weighted average.
Both of these things are actually true. Attention is indeed "just" a weighted average (there is a proof in the paper in my original reply), AND ALSO, the Q/K are learned, and the similarity between two learned vectors suggests there is more going on than might initially be obvious. (I actually argue in the top comment that the positional information is the #1 reason why attention works, so I agree with this.)
Let's see why.
What are Q and K? The idea in the original paper was: we have some learned embeddings for our tokens, and some "context" we'd like to use to enrich those embeddings. Stuff like the positional information, context accumulated through layers, other learned stuff. So we do this enrichment, and that produces the so-called "query" (Q) and a "key" (K) vectors. Attention is STILL a weighted average, where the weights are determined by how "similar" those vectors are (this is what we're talking about when we discuss the "distance" provided by the so-called "kernel function").
Anyway the reason it's hard for me to parse this reply is because I think a lot is getting lost in the transition from a second language to English. For example, "distance" in this context means something specific, and to be precise the exponential kernel provides a similarity measurement (dot product can encode distance but generally requires fixed-length norms). Similarly, I am confused by their use of "Euclidean" distance (kernel functions do not require this, see p. 203 of the seminal Elements of Statistical Learning which provides a general construction of anisotropic distance-based kernels which is well-accepted), I'm not sure how to interpret "what you feed to the kernel is Euclidean distance" (a pairwise kernel should compute the similarity, not be fed the similarity) and so on. None of this is the end of the world, I am just having a very hard time figuring out what they meant.
[1]: https://mybiostats.files.wordpress.com/2015/03/chapter5-kern...