____
I agree that that's a great strategy in general, but I don't think it translates well to display servers.
You can't just "put it in a sandbox". You need driver support, and you'd need a sandboxed environment sophisticated enough to deal with all the idiosyncrasies of the X protocol.
Even putting running an old version of X11/UNIX in a virtual machine is of limited benefit, because X11's core functionality is often tightly coupled with drivers and other things which would need a lot of special wrappers and hacks to work properly.
I'm not saying it wouldn't work (and I am not an X11 developer), but I do think there are extremely compelling and legitimate reasons for people to not follow your suggestion (at least in this case).