It is different models: X11 assume you have only one terminal (effectively client, but "server" in X11 parlance) and many system to run software (effectively servers, but "client" in X11 parlance).
VNC/RDP works other way around: one server, which runs all you programs, and you can attach to it from different terminals/clients.
It is not exactly so, as RDP can forward only some programs (windows) and not full screen, of course, but close enough.
I cannot speak for everybody, but for me (and my friends with whom I discussed this) second use case is much more important, than first one.
About speed: maybe, with 10Mbit Ethernet it was true. But now I could work with Photoshop (!) from my office (client) on my home workstation (server) via RDP and don't notice delays. Yes, I have 1000/100Mbit asymmetric connection at home and I don't know what is used by my office. One time I've forgot that it is RDP to my home sysmtem and started Youtube video. I was surprised, that video and sound is slightly off-sync, and only after that recognized that it is RDP, not local browser!
Much worse connection is enough for less demanding tasks, I've worked with "normal" not graphics-heavy programs via 4G connection in India (my home system is in the Netherlands) and it was not painful. Yes, there was perceivable delay, but for task like "Open PDF, open browser, fill form on site by copy-n-pasting strings from PDF, submitting form and authorize with 2FA from phone" it was perfectly Ok.
But, yes, if you need to assemble 10 windows from 10 remote machines on one screen, X11 is the best.
RDP actually delivers on the promise of local-feeling remote apps.
(The revenge of the UNIX-HATERS is that Microsoft designed a better shell than sh (PowerShell), a better X than X (RDP), and a better Emacs than Emacs (Visual Studio Code).)
VNC really sucked on the other hand, not being able to transparently share single windows (at least I never figured out how to), some windows would fail to refresh and I would need to drag them around to get them to redraw, copy and paste was always a pain, sometimes inputs not registering properly.
To be fair to VNC though, over the internet plain X11 forwarding really sucked (latency is the real killer here) and VNC won out there. Unless one was using NX proxy, then it blew VNC out of the water (while using X11 on both ends). Only RDP was somewhat on par with NX over the internet, but locally still beaten by X11.
And for me inability to detach whole session from one server (in X11 terms) and attach to another is showstopper.
X11 needed something like console "screen"/"tmux" from the very beginning, in the core protocol, IMHO. Not for multiplexing of workspaces or desktops, but for this session re-attaching.