Obviously. The breakthrough was called “attention is all you need”! self-attention was being studied as an approach, like alternate schedulers, learning rates, various architecture connectivies, and so on. It turns out self-attention happened to include all of that; that’s the insight.