I think that if you do things correctly, your concern is mostly mitigated. My last job was Java/Spring. As someone noted above, you really don't want to be working with Java/Spring without IntelliJ.
But we never had issues with anyone breaking builds due to screwing up project files, because those weren't shared and the project would run/build/test immediately after import into the IDE without any additional tweaks.
I think it's sort of odd that many folks complain about IDEs, but then use vim with a million plugins. Isn't vim + plugins = IDE? Or, if the overall point is "java sucks, don't use java" then fine, but if you have to use it, I couldn't even imagine using plain old vim without any language support...