But we're not talking about the same thing here. I do also work 99% in the terminal, and SSH is perfectly okay.

But my window manager is running locally.

In a thin client/remote desktop setup the WM is running remotely. From my experience, this makes the mouse and keyboard ever so slightly laggy that's it's very annoying.