X is dependent on you using a Operating system supporting X. Ssh can be used across all operating systems, even as an Android app. Sometimes that can be useful.
I'm not arguing that X11 is more streamlined than remote SSH (or tmux!) in a terminal / console. But it is far more available than you might think.
You probably also refer to Windows and MacOS, but that just corresponds to another type of difficulty (Xming, Xquartz), it doesn't make it inherently worse/impossible.