There were X11 extensions that implemented access controls, eg in TrustedSolaris.