I require both key and password to login, and have fail2ban rate limit password attempts.

This gives sufficient notice to fix things if a key were to become compromised.