Comment by bmer | Hacker News Reader

Comment by bmer | Hacker News Reader

I'm not sure if it's just me, but this feels like a terrible article. It simply does not explain why "self-attention" is called self-attention in the first place, which would be very helpful for building intuition. Luckily, Wikipedia at least explains why "attention" techniques are labelled as "attention":

> In the context of neural networks, attention is a technique that mimics cognitive attention. The effect enhances the important parts of the input data and fades out the rest—the thought being that the network should devote more computing power to that small but important part of the data. Which part of the data is more important than others depends on the context and is learned through training data by gradient descent.

Now, it remains to figure out what the "self" if "self-attention" is for, but maybe someone else can fill that gap. ML isn't something I care about particularly, so don't want to spend too much time.

It's called self attention because you only look at the input itself to determine which parts of the input to focus on. Transformers where first proposed for machine translation, and the neural net used in that publication consisted of two parts: an "encoder" part that takes an input language sentence and uses this self-attention to better understand it (and "encode" it into some neural representation), and a decoder which takes both the encoded source-language input as well as already translated parts of the target level sentence as input. These encoders use cross attention, ie they use the input sentence to determine on which part of the output to focus on next.

But that sounds exactly like the memory in long/short term memory recurrent neural networks (LSTM RNN)

The key difference is that RNNs store the already seen parts of the input to use it when processing future inputs, while self attention allows you to focus on all parts in parallel: information from the last input token can be used on the first token, which in RNNs is not possible (NB: that's one of many differences, but likely the one most relevant in this context)

Reply on news.ycombinator.com