You misread my post, what I meant was that your day-to-day account is vulnerable, since you run your applications with it. If that's the account belonging to wheel, once your user account is compromised the attacker is root as well. One should use a different account, hopefully harder to compromise since is used only for sudo.

That said, I like your use of yubikeys, even better than the phone that's the sort of stuff that should be triggered using a smartwatch.