What makes it approximately permutation equivariant (vs entirely)? As I understand things, if the order is jumbled, the attention matrix does get its rows and cols permuted in the way you'd expect so I'd have thought they'd be entirely permutation equivariant.

I see, so [1, 2, 3, 4, 5, 6, 7, 8] is more similar to [5, 3, 1, 7, 8, 2, 4, 6] than with a linear layer? That's what you mean by permutation equivariant?

I understand each context input is embedded with its position, but I suppose the transformer can learn to ignore the position and just look at the context as an unordered set?