I'm not an expert in this, so hopefully someone more knowledgeable can weight in - but SVMs are understood much better from the perspective over overfitting and things like the VC bound - while Transformers are not really understood as well. From what I remember it's quite easy to have a SVM overfit, while Transformers have fewer issues. It'd be interesting to understand why

So if the two are somehow connected, then that could have implications for tuning and fighting overfitting

maybe it'd also be possible to design better non-overfitting SVMs