I have ~2ms input latency using vim on xterm. I doubt VSCode can ever achieve this kind of performance, simply due to the poor architectural decision of using electron and JavaScript.
this does not really strike me as odd for emacs and vim - pretty sure that if you load them in a computer with a 640 by 480 screen in a 32-bit OS on IceWM it will be comparable to the olden days. On todays's 4k screens, just opening a window that takes half of your screen will alone cost a good dozen megabytes of RAM - and if your UI widgets do caching (they should, text rendering is expensive), then each of your widgets may allocate a pixmap too. When screens were 72 DPI and a button was 30 by 10 pixels it was fine... but nowadays you have to multiply all the values by at least 4.
And the switch to 64bit has instantaneously doubled the size of all the data structures working with pointers - which is generally a lot in GUI apps since most are trees of widgets. In some of the apps I've been working on, the RAM hit of 64bit has been more than 30%, and sadly the world does not seem to accept the x32 ABI.
And voilà, you can have your 32-bit pointers. You can even go down to 16-bits for small enough arenas.
(No, I'm not actually serious. Though we could imagine something similar on some new language.)
We can also have them with zero hacking since there is an existing ABI for 32 bits pointers in 64 bit mode: https://wikipedia.org/wiki/X32_ABI ; you get all the benefits of x86_64 mode (more registers, default support of SSE2 etc) for the memory usage level of x86 (and 4gb of accessible ram instead of x86 OSes's 2.something addressable gigabytes). I'm pretty sure that only specialized apps actually require >4gb of ram - most common utilities (e.g. shell, text editor, etc...) should really be compiled in x32 mode.
That's somewhat specialized, but not as much as you might think.