If you want that flexibility and power, the options must live somewhere. Dividing that into two mutually exclusive modes breaks up the complexity of the main interface. I need to know a handful of key combinations like ESC, i, r, h, j, k, l (or arrow keys on modern systems), :, w, q, s, d, g, G, v, V, /, ?, !, % to get a vast majority of daily tasks done. How many nested drop-down menus would I need to navigate or multi-modifier hotkeys must I remember to do the same thing just so I don't have to remember one bit (which mode I'm in) before hitting a key?
The human mind is actually very comfortable with modes. Work, home, visiting family, visiting friends. Cooking at home vs dining out vs ordering in vs baking vs grilling vs cooking a holiday feast vs catering in vs party snacks and drinks vs road trip snacks. Driving vs taxi vs riding a bike vs walking. Retail vs garage sales vs in-person auctions vs online auctions vs online fixed-price shopping. Lending libraries vs bookstores vs reading rooms vs newsstands vs books we own vs ebooks. Math vs prose vs poetry vs visual communication vs music. We pay attention to different things and follow different sets of rules in different circumstances all the time. Is it easier to remember all the rules for all scenarios at once or to switch contexts?