I'm double checking it right now using a remove machine a few blocks away from my home. Running `ssh -X remote firefox` lags a lot worse than `waypipe ssh remote firefox`. The latter feels almost native, very responsive.
So no, I don't think we are giving up a lot. X11 was designed to work over network, but it never really did.