Company mode is the big offender for me, it handles very large completion lists poorly. I'm not yet fully satisfied with the latency situation in my emacs config, but for me there is no going back. Evil mode let me keep almost all of the muscle memory I ever learned in vim/neovim, and the rest of the emacs ecosystem more than makes up for the latency issues in my opinion.

Honestly this is a big thing that emacs folks don't value, multi threaded performance.

I don't even use LSP because I have my own ways of browsing source code through emacs. So I go off and tout emacs' amazingness, but then when someone tries it and attempts to use what worked in other editors, it's a slow laggy mess.