> If we want to move to a version with minimal privileges
Then you would have to cut everything provided by the X protocol into many, smaller, controllable pieces. Because if the permission control is just a switch that says "ability to communicate with the X server", then the whole exercise is rather moot, isn't it.
And when you cut the protocol into smaller pieces...