Hacker News Reader

transformers were also just better at the LM task than 2018 RNNs for equal amount of flop training

Yeah, that's just the training stability part to my knowledge

they're also just less capable models. like just adding attention on top of an RNN made them a lot better

Calculating self-attention is still quadratic though. So you get the negatives of transformers there too.