Maybe people have extremely fancy shell setups? Perhaps it's a font thing that slows it down? Can you demonstrate?
I'm usual a tmux/zsh/vim user (although I've promised to switch to emacs for years :( )
I just timed my xterm. I'm displaying about 2.3MB/s of characters.
Basically here was my test
$ dd if=/dev/urandom bs=10M count=1 | hexdump -C > /tmp/test
$ ls -l /tmp/test
-rw-r--r-- 1 chris chris 51773449 Oct 8 09:32 /tmp/test
$ time cat /tmp/test
... wait wait wait ....
51773449 / (total time) / (2^20)
Now this of course depends on the size of the window. I'm on a 4320x3840 dual 4k monitor and the window geometry was 1334x928 for this test.I acknowledge this computer (i7-8650U ~ 2018 thinkpad with a boring UHD Graphics 620) is capable of faster output but the xterm featureset is vaaassstt (albeit extremely complicated) and I've found the other terminals to be lacking.
My only faithless move away from xterm was probably 17 or so years ago with rxvt when I was on a laptop with ballpark 16MB of memory. It's memory footprint and startup time was way smaller probably for its total lack of features and unicode support. That made a difference on my Toshiba Protege 120mhz pentium I was using at the time.
When I was a macOS and a Windows developer, slow terminals were definitely a problem though
Casey Muratori demonstrated this with his refterm project. It seems like a lot of interest in making fast terminals has followed from that.
This isn't rhetorical. I can't answer the question
TL;DR refterm (while positively inspirational to a few) was harmful to the terminal community, because it was fooling a lot of people with false metrics that the users simply believed in, without questioning.