> but the penalty is that they need lots of training data and compute.

I think it depends on how noisy and unstructured data is (e.g. human language), transformer will likely approximate few well correlated datapoints just fine.