The term "attention" confuses more than it reveals, trading well-known terminology for a weak analogy to how the human brain works. Kernel smoothing is all you need.
Imagine you have a series of (x,y) coordinates. You receive a new x coordinate, and you'd like to predict the corresponding y. So, the absolute dumbest thing you can do is find the nearest x, and then report its corresponding y as your prediction. The second dumbest thing you can do is find the n closest x coordinates, and then average their y values together, and output that as your predicted value. And the third-dumbest thing you can do is that n-closest-average thing, but give the "closest" x coordinates the most "weight" when you average them. (Technically, not always "closest": you have some "kernel" function which does the weighting, but "closeness" is a good kernel so I will use it here.)
That third-dumbest thing is Nadaraya-Watson kernel smoothing. And that's what attention does. You have a big list of vectors. You compute a weighted average of those vectors. You now have a new vector. You can do what you want with it, but LLMs will generally turn that into a distribution over token types, from which you'll generally want to select either the top one, or weighted-randomly one from the top k (known as "increasing the temperature").
With all that said, I'm not sure "Kernel smoothing is all you need" is actually materially better than "attention is all you need" for 99.999% of humans. :)
https://scikit-learn.org/stable/modules/generated/sklearn.im...
https://scikit-learn.org/stable/modules/generated/sklearn.me...
Like an accountant not realizing that addition and multiplication are special cases of a monoid operation.
Aside from that, kernel smoothing has roughly 70 years of literature devoted to every practical aspect you could imagine: approximation, accuracy, pedagogy, behavior under sparsity, relationship to other well-known theories like Bayeisan learning, and so on. Variants like Performer and elliptical attention, all directly arose from the knowledge that attention is intro-grad-stats kernel smoothing. A large portion of our understanding of ICL is directly or indirectly resultant from the fact of this correspondence.
I don't have any specific experience in ML but I always liked the formalism presented in https://www.youtube.com/watch?v=J1YCdVogd14 as a way to unify everything
Now if you could please explain the rest of math in similarly simple terms?
btw, "kernel smoothing" etc. are still fairly complex terms that sound a bit like math magic: 1-nearest-neighbor (1-nn), k-nearest-neighbor (knn), and weighted knn would be even simpler and possibly more descriptive.
First of all, the trick relies on imputed 1NN/imputed kNN/imputed weighted kNN. And second of all, you still have to explain how imputed weighted kNN relates to the problem of taking a bunch of x/y pairs and predicting a novel y for some novel x. When you start to unpack all of that it does not look simpler to me than what I wrote.
For the weights, I'm not sure that's as mysterious as you make it seem. The most important thing they learn is, basically syntax. So you have 40k token types to choose from at each prediction, and the vast majority of them are statistically implausible because they are syntactically invalid. At some positions this restriction reduces entropy from ~15-16 bits (or 40k choices) to 2^8 (~200 choices) and in some places it reduces it "only" to 12 or 13 (~8k choices, common when the next token is "approximately a noun").
Once you winnow it down, it's much closer to a moderately big multiple choice problem. And much of that entropy lies just in the model's representation of word co-occurrences.
First and most importantly, attention is, formally, kernel smoothing with a particular exponential kernel function, and if you don't believe me, you can read the paper I linked, which proves it.
Second, and less importantly: kernel smoothing absolutely, definitely does NOT require a kernel function representing Euclidean distance, and in fact the entire point of kernel smoothing is that you can pick basically any function that fits your needs. Many use cases find it helpful to have a real-valued kernel function that is positive and semidefinite, or nonnegative and integrating to 1, or whatever. Different kernel functions in these spaces have different properties and trade-offs. The exponential kernel attention uses (if memory serves) is (1) finite and nonnegative, and (2) has a positive total weight so you can normalize.
Third, and least importantly, you definitely can have a kernel function that is "uniform" (i.e., "just" an average) although it's generally not very helpful (e.g., not positive semidefinite).
Well, here's the thing. Kernel smoothing uses Euclidean distances. "with a particular exponential kernel function" is the novel bit. I even gave an example of what absurdity you will get if you allow arbitrary addition like this to the term.
> every assertion here is mistaken. ... Second, ..
Never made that assertion.
> Many use cases find it helpful to have a real-valued kernel function that is positive and semidefinite, or nonnegative and integrating to 1, or whatever
Yeah, that's the kernel. https://en.wikipedia.org/wiki/Kernel_(statistics). What you feed to the kernel is Euclidean distance. That's where the difference is.
> you definitely can have a kernel function that is "uniform"
You did not get that point. See response to your "First". I was explaining why the reasoning that led you to "attention == kernel smoothing" is faulty - if you apply it to averaging, you will get "averaging == kernel smoothing". Because kernel smoothing is "just" averaging with distance based weights. So attention is "just" kernel smoothing with Q-K based distance function. The issue is in both cases the part after "with " does all the heavy lifting.
For those following along, when this user says that attention is kernel smoothing with a "Q-K based distance function", I think (THINK!) what they're trying to say is the second part does more work than it gets credit for when we say attention is "just" a weighted average.
Both of these things are actually true. Attention is indeed "just" a weighted average (there is a proof in the paper in my original reply), AND ALSO, the Q/K are learned, and the similarity between two learned vectors suggests there is more going on than might initially be obvious. (I actually argue in the top comment that the positional information is the #1 reason why attention works, so I agree with this.)
Let's see why.
What are Q and K? The idea in the original paper was: we have some learned embeddings for our tokens, and some "context" we'd like to use to enrich those embeddings. Stuff like the positional information, context accumulated through layers, other learned stuff. So we do this enrichment, and that produces the so-called "query" (Q) and a "key" (K) vectors. Attention is STILL a weighted average, where the weights are determined by how "similar" those vectors are (this is what we're talking about when we discuss the "distance" provided by the so-called "kernel function").
Anyway the reason it's hard for me to parse this reply is because I think a lot is getting lost in the transition from a second language to English. For example, "distance" in this context means something specific, and to be precise the exponential kernel provides a similarity measurement (dot product can encode distance but generally requires fixed-length norms). Similarly, I am confused by their use of "Euclidean" distance (kernel functions do not require this, see p. 203 of the seminal Elements of Statistical Learning which provides a general construction of anisotropic distance-based kernels which is well-accepted), I'm not sure how to interpret "what you feed to the kernel is Euclidean distance" (a pairwise kernel should compute the similarity, not be fed the similarity) and so on. None of this is the end of the world, I am just having a very hard time figuring out what they meant.
[1]: https://mybiostats.files.wordpress.com/2015/03/chapter5-kern...
Are you, in essence, implying that the Key-Query attention mechanism is basically a neural function approximation of the Nadaraya-Watson kernel or something very like it?
I thought both were useful in their own way, it's not like one replaced the other right?