The attacker can install a keylogger as normal user and then wait for the sudo password. How much better is that, really?
Almost no one has the resources to stand up to a targeted attack from even a mildly capable hacker. Therefore, the risk profile most administrators are working against are attacks of opportunity. In that race, as in the old joke about outrunning your friend instead of a lion, you need only be be some small amount more difficult to crack than the next potential victim.