RDP also does its fair share of raster shuffling. Where it has the edge is that most windows programs still use GDI (i.e. windows native draw commands), and RDP can send a lot of that as commands instead of pixels.

Xwindows used to have the same advantage, but lost it for most modern applications (i.e. ones written in GTK or Qt, which pre-rasterize almost everything for simplified cross-platform compatibility). Nowadays, unless you restrict yourself to classic X applications, X forwarding is going to be a slower, dumber, version of VNC.

Mac has so many fades & animations that, raster or command, it will probably always be slower.