The problem I see is: a lot of things that non-modal, GUI interfaces use mouse for can be better done with a keyboard.
For example, all sorts of drop-down menus and icons that you click on to launch programs. This kind of interface is inherently synchronous: the computer suggests, you pick. To me, it's way more natural to asyncronously type in what I want without having my attention hijacked by whatever suggestions my machine has (sometimes those actually make me forget what I was trying to do in the first place). I love the related metaphor ESR came up with in his Unix Coans [1].
[1]: http://www.catb.org/~esr/writings/unix-koans/gui-programmer....