I haven't dug through the X server code, but byte-swapping ought to be an insignificant attack surface of the code. Most network protocols do this already; ssh does it!
The only thing that matches with what I have observed from the X developers is that the X code is, in general, a steaming pile of shit that nobody wants to wade through anymore and so any bit that can be disabled and/or removed is unambiguously a Good Thing. The design of Wayland strongly implies that network transparency is a non-feature for the X developers, so everything surrounding it will be a target for removal.