Personally I don't have passwordless sudo but I'm also curious as to what the attack vector might be here if you were to enable it.
Basically, if you have malware running as your own user, I'd be surprised if it couldn't find a way to trick you into typing your password to invoke sudo. Even then, it can probably do quite a bit of damage without sudo.
So then the only scenario a sudo password could possibly save me from is when I have malware already running as my user (and it's unlikely that it would help anyway due to the tricks I just mentioned). Of course everyone should do what they think is best, but personally if I were in this scenario it's pretty much game over anyway, so it's not something I'm going to worry about.