Losing Shift+F6 refactoring is a little annoying, but definitely not something that makes programming Java impossible. Autocomplete might get a little hairy as well if you're referring to classes that exist in Jar files too, or am I wrong about this?
As for everything else, using the CLI for source control and building is something I do anyways, even when using an IDE. At least with Gradle, that is. As long as some of those issues have solutions, though, I think this might be possible. Maybe I'll have look into it a bit more.