Yeah, I also use VIM bindings in VS Code and love it. When I'm at my desk, in the zone, writing code, my hands pretty much don't leave the keyboard. But when I'm leaning back, browsing through files, etc. I just need to press 'i' once and I have a "normal" IDE.
I couldn't imagine not being able to hover my cursor over something to read its type definitions, cmd+click to go into whatever I'm clicking on, etc.
Sometimes I see one of those videos of a fully tricked out NeoVIM and it looks tempting, but I know I wouldn't actually like it.