It's not vim, but the OS/graphical system that you are most likely using underneath. While having Firefox/Chromium open for reference to what you are working on.
Specifically worth comparing terminal emulator performance. I noticed that for example (not surprisingly I guess) alacritty has been both the worst and the best depending on the setup.