Only after learning vim you realize how clumsy and slow this is.
On the other hand, having to constantly hold Control/Alt, including long chords (e.g. with prefix + count), feels much worse for me. I use Emacs keybinds on the terminal (outside of vim), but I can't imagine having to constantly press modifiers. It's not as comfortable as switching to a mode where those actions are explictly first-class.