It's not just the internals of X11 that are the problem, it's that the entire concept was baked in a different time. The major innovation of the X11 server was that, well, it's a server. It can be completely decoupled from the client, even physically if you use it over tcp. That was kinda nice, but then came GPUs, and direct rendering. The name of the game is now violating every single abstraction that stands between a userland app drawing directly to the display with only a GPU and a single graphics api in between. It was definitely not junior developers who rewrote the entire kernel graphics stack to move from frame buffers to direct rendering.
Computer graphics are pretty tough if you think about how many bits/s are being transferred to your 4k display at 60hz and how little latency we will tolerate. It's not like network data where we can afford to encapsulate it five times over and abstract it and forget about the contents.
Any criticism of Wayland should be with regard to what is next. Hell I'm not even convinced we need a display server, let's talk about that. I'm not going to take anyone seriously who says X11 is better, and because of a few insignificant issues like taking a damn screen cap (really? and that works fine now btw).