Yes! Probably my biggest workflow pipe dream since getting some fluency in vim is to be able to drop into an "outside mode". I've cobbled something vaguely modal together for bspwm with abuse of shkd, so a lot of the binds map similarly, but like your solution it's far from ideal.

As for "Why would you want this?" For me, the straightforward answer is "modes are modular". A core concept in vim is that your input can be thought of as a string describing the edit you wish to make, and modes help me keep that manageable in my head: I can first think about the mode I plan to enter, and then separately what I plan to do under the constraints of that mode. It's a little hard to explain, and I'm sure acolytes of other paradigms have their own philosophies, but it fits in really well to my workflow.

It's a subtle thing in the limited world of text editing, but I definitely miss the semantics whenever I leave it, whether to a terminal or a browser or whatever. So for me, it comes down to wanting to bring the cost of task switching closer to what it is in the editor.