Check out Onivim 2 as well for a VSCode like editor that natively uses Vim, but can use VSCode plugins and is generally easier to use than Vim. It also isn't built on Electron so it's super fast and lightweight. Currently it's in beta I think.
Essentially, to me it's all about ergonomics of editing. Hands straight on the "home row" of the keyboard, and with modal editing, almost everything is a single keypress away. To me, this feels a lot faster and easier than reaching for the mouse (or arrow keys) every five seconds, and I get the additional benefit that my core editing commands stay consistent across different environments (where I have a Vim plugin, that is). I leave all the fancier, IDE-specific stuff (e.g. refactoring, etc) on their default IDE bindings, and learn them. It's a good compromise, I'm not sure why people don't like it.
Your experience mirrors mine in that most of the benefit of vim is in the basic motions and modal way of editing, which most of the plugins provide in a decent enough way for me.