> predict a weight of exactly zero for some of the values
Wouldn’t this be pretty unlikely, though?
Wouldn’t this be pretty unlikely, though?
Attention is really good at finding this smattering of words (ie assign most weight there). But it struggles to put exactly 0 on the other words.
SoftmaxA[n] - SoftmaxB[n] is exactly 0?
Even if 2 attention layers learn two different things, I would imagine the corresponding weights in each layer wouldn’t exactly cancel each other out.