You can enjoy the best of both worlds with something like zsh's edit-command-line. I'm very happy with Emacs keys in the shell, unless I need to make a non-trivial edit or type in a multi-line command. In those cases, C-x C-e drops me right into Vim with the current command-line ready for editing.