Furthermore. transformer xl extends the convolutional architecture to use some recurrence. I think that claims that RNNs have been wholesale replaced by convolutions are premature.
I can't speak for 'gwern, but I was also under the impression that CNNs had basically replaced RNNs for these tasks. I think my view is colored by Jeremy Howard's FastAI talks. Since he isn't big on publishing papers (though that is changing a bit via collaborations with his students), it may be true that the state of the art is no longer best represented by published papers. (I would argue that avoiding this ambiguity is an argument in favor of publication.)
It’s common to use CNNs in practice because they get you most of the way and are so much cheaper to train/run. They are, however, limited in context to the width of the receptive field.