When I first learned about X forwarding we had a modem, and X was responsive. I also played CivII over remote desktop and it was...less responsive.

With Wayland, have you tried out waypipe yet?