Cosma Shalizi explains it very well.
"Again: Calling this "attention" at best a joke."
http://bactra.org/notebooks/nn-attention-and-transformers.ht...
"Again: Calling this "attention" at best a joke."
http://bactra.org/notebooks/nn-attention-and-transformers.ht...
Anyway just curious do you or someone else have more sources in this format?
At least this way it is very helpfull to think about the methods. Kind of agree that the formular does look very very similar to well known methods. But on the other hand, the author doesn't explain the transformers in a similar simple way since it is not obvious why you could stack kernels and get better results.