Eh even if you secure the X11 API itself, your isolated app (browsers absolutely sandbox and isolate themselves from the rest of the system) will still share memory and have a socket open to an 33 year old c codebase (XOrg).
I think the Qubes approach is the only one worth considering if one deeply cares about security.