Anyone who finds this paper interesting should read: "self attention between datapoints" Which shows non-parametric transformers often beat GBDTs on tabular data.

:https://www.google.com/url?sa=t&source=web&rct=j&url=https:/...