Watch a skilled Dota player in action, you will find a lot of muscle memory with the mouse there.
Now compare this with an usual "desktop session", where you may have multiple windows which aren't always in the exact same place, the cursor moves around, maybe even hidden because you've been typing, etc. Now you have to spend some extra time to 1. locate the cursor, 2. locate the button to click, 3. move the mouse over. It will probably not be slow but it clearly takes you out of the flow compared to keeping your hands where they were and knowing that whatever your shortcut you have to press will do what you expect.
What you say reminds me of synchronous-asynchronous analogy I just made in another branch of this thread.