X11 is also just a protocol and you can implement it securely and expose access control through extensions. In fact it already exists (XACE) just nobody is using it.
https://www.freedesktop.org/software/XDevConf/x-security-wal...
In Wayland, you make the extension, then you patch every compositor, every toolkit and every application.
In X11, you make the extension, then you patch the X server, Xlib, XCB, every window manager, every toolkit, and every application.