Not entirely true. Now if you want to share the "display server" bits you can plug in a library like wlroots or Smithay. The real issue, if any, is that most of those bits are not standardized; or at least, only the libwayland part.
Of course, that's not even the entire story either. When X11 servers were basically entire operating systems that ran as root, implemented print servers and directly touched /dev/mem, only a lunatic would want to maintain multiple of them... but now on Linux and even some BSDs, you can fall back to GBM/DRM/KMS/libinput/etc. for most of your "hardware" needs, the OS is doing the OS things. Wayland is also just simpler than X11; The current day cleaned-up X.org is still around 5x larger by SLOC and much more complex than wlroots.
So how come GNOME and KDE both built their own display server? Well really, they didn't; they took their existing compositors they had for X11 anyway and grew Wayland server limbs out of them. In practice retrofitting Wayland into existing compositors is still hard, but if you already have written a compositor and window manager it's still probably easier than writing a new one from scratch. And for KWin, I believe they're also making use of Qt's Wayland compositor infrastructure, too.
Something that took me a long time to understand is that people are just unhappy no matter what. Not that all of their points are invalid, just, it's often not even really about that; the specific points are often just the obvious weak spots, like the classic "Wayland can't screen capture" meme (totally fair point, but persisted so long that some people still don't know it hasn't been an issue for years.) This right here is just another one of those. Notice that when many Linux distros unified towards a singular init system, systemd, people came out with pitchforks. Were they all spouting incorrect nonsense? Well, blah blah UNIX philosophy, but largely no, there were plenty of valid complaints about systemd and what it meant for the Linux ecosystem, but it wasn't really about that. Same with Wayland! Wayland did the exact opposite, and defined a set of protocols anyone could implement, allowing you to choose your compositor. Seeing as people hated what systemd did and wanted to preserve "choice" in the Linux world, you'd expect that people would like what Wayland did, but instead everyone is just pissed about fragmentation now.
You can pick apart that argument and try to figure out why it's wrong or and invalid comparison and I'm not going to try to fight anyone on that. You can also say that the argument is made by different people and maybe that's true too (though personally I believe "people who hated systemd" and "people who hate Wayland" forms more of a circle than a venn diagram.) Even if it is, the outcome is the same: you're damned if you do and damned if you don't. What I do know is that desktops on X11 stagnated with the same sorts of problems they always had and with Wayland a lot of that got unblocked. X11 will probably be in the same crappy state it's in today and Wayland will continue iterating and improving over time. I'm sure there will be a lot of pain along the way, and I agree it isn't always pretty, but it is what it is. Software isn't a beauty contest, and there's a good reason why "elegant" stuff rarely wins in the long run.