Did you mean fast in terms of performance, or in terms of your ability to edit, or something else? And you weren't using a GUI version of vim/emacs, right?
So scrolling is slow. Terminal related? Maybe - terminals are slow. But when highlights an error in elm or ocaml code, scrolling becomes unusably slow (not that it's really that fast otherwise).
By contrast, no plugin I loaded made VS Code slow, and it was super smooth and fast to do, yes, scrolling, all the time.
My current emacs is via spacemacs, and I'm only using it for magit. Apart from asking me to update .recentf EVERY SINGLE TIME I refresh magit, it also takes a very noticeable pause to update the git status. By contrast, that was completely backgrounded in VS Code (though I didn't really use it since I prefer magit, so apples and oranges comparison here).
TL;DR: vim is not fast, especially not if you want to do something like use it in a terminal with plugins. VS Code is actually really fast.
From Alacritty's FAQ: Q: "macOS + tmux + vim is slow! I thought this was supposed to be fast! This appears to be an issue outside of terminal emulators; either macOS has an IPC performance issue, or either tmux or vim (or both) have a bug. This same issue can be seen in iTerm2 and Terminal.app. I've found that if tmux is running on another machine which is connected to Alacritty via SSH, this issue disappears. Actual throughput and rendering performance are still better in Alacritty."
With regards to vim though, there is no reason for it to be slow, except for some specific plugin pulling you down. Are you using synchronous linters? I thought Ale was async. I would also recommend vim-plug and lazy loading for plugin management.
I’m on Mac myself and use vim regularly in iTerm, Terminal.app and Gui, and am currently not experiencing any problems. Of course, environments vary.
Also, you can try switching out your vim with neovim, which should work seamlessly.
The worst is probably language specific support for LaTeX files, particularly syntax highlighting, there is even a tex-slow (http://vimdoc.sourceforge.net/htmldoc/syntax.html#tex-slow) entry in the Vim manual. On my old netbook (haven't heard that word in years!) it was so bad Vim lagged behind my typing, and I'm not particularly fast. I turned off syntax highlighting which fixed the lag, but missed it so much I switched to Emacs instead (and discovered AucTeX which is fantastic, so I wouldn't switch back now).
I'm no Vim or Mac OS poweruser though - I've just kinda put up with or avoided it. Sorry I couldn't really offer much beyond confirmation :)