A large part of the reason why that's so annoying is that when you do that you have to also figure out where the mouse pointer is, and then start the mouse moving in the right direction. Once the mouse is moving the interaction is very natural and you can do it by auto-pilot. But until it is, you actually have to use a little piece of your brain which you'd rather focus on the problem at hand.

Touch should completely negate that. Just touch what you're looking at; you should be able to do that without requiring any higher brain activity.