I would argue that if you sum the code of all the Wayland compositors existing today + all the dependencies required for making them work as a proper desktop - dbus, wlroots, ... you likely have much more code (and thus much more bugs overall) than you had for X11.
xorg git repo is 400kloc of C, I've seen projects which were on the order of magnitude of 40 million lines of code and still were able to be managed, so to me "sheer code size" sounds a bit weird. Hell, the main project I work on, https://ossia.io and on which I do the huge majority of the swdev work alone is nearing 500kloc and it does not feel like a burden.