What exactly do you think are the problems in X11 core protocol and extension mechanism that required to start from scratch?
Broadly, the X Server has a bunch of capabilities which are irrelevant. The modern model is really Window <-> Compositor based, and the X Server protocol is just a pointless middle man in that exchange.
"compatibilities which are irrelevant" do exist, such as old drawing primitives but those are not really an issue. They can be maintained for backwards compatibility and eventually deprecated and removed, but they are not anything which holds back modern clients. The X server protocol is not anymore a "pointless middle man in that exchange" than the Wayland protocol is a "pointless middle man". A protocol is obviously needed so can not be pointless.