> If you disable passwords, then you need to use password-less sudo.
unfortunately that's true. But it shouldn't have to be. There is a libpam-ssh-agent-auth which makes use of a forwarded ssh-agent socket to authenticate to whenever sudo needs it, which means that you have to re-authenticate your ssh-key locally.
It works, but it never really took of, and tbh I wouldn't trust it's security. But the authentication flow is something I would want.