Eh even if you secure the X11 API itself, the Xorg server still is a 33 year old (!) c codebase.
It is mostly "secure" due to it being used in practically every server and billions of devices, so there is an active maintainer community around it. Xorg has none of that.