A few of my colleagues are vim users, and fortunately for the others we are not as vocal as you describe. I already accepted of being the weirdo that uses vim, because my programming skills make up for it ;).
I do have a shortcut to disable it, when we do some pair programming.
How is that these days? I used it a couple of years ago but it had issues at the time, namely some performance problems and getting stuck in a particular mode.
I ended up abandoning vim bindings because the JetBrains plugin was also buggy, and I straight up had no feasible way to get those bindings in Xcode, but I hold out hope for a glorious LSP-based future where I can just use vim and get all the benefits of my other editors.
I tried using VSCode with the Vim plugin for a while but the latency and inconsistencies eventually got me back to proper (neo)vim. UIs are almost always more complex and subtle than people intuit, have a large surface area, and make lag and inconsistencies most noticeable.
I love it, and using anything else feels weird. I remember having some problems with it before as well, but I haven't bothered looking at the settings or anything for months. I had to go back just now to see if I had any weird configs set up, (I don't)