X11 does have various ways to restrict access (one of which ssh does use for instance) and some more advanced security extensions. But as far as I can tell there has never been that much motivation to widely deploy any of it.
It is mostly "secure" due to it being used in practically every server and billions of devices, so there is an active maintainer community around it. Xorg has none of that.