I used vim for 5 years, afterwards I switched over to Emacs, and I couldn't disagree with you more. What's clumsy and slow is having to switch between insert mode, normal mode, and visual mode hundreds of times while working rather than just using modifier keys for special actions.