An SSH private key is almost impossible to brute force, compared to passwords. (unless your SSH key generator has been patched by a clueless Debian maintainer)
I set the root password (over ssh) to something random but only 8 characters. I meant to change it to use only keys, of course, but the machine was compromised in under an hour!
(Granted, this guide specifies that it's meant for sysadmins managing groups of workstations, in which case SSH access might be necessary for remote administration, but for most users, SSHing into a workstation is unnecessary.)
I've also remoted into my workstation from home to do emergency work.
SSHd allows you to restrict particular users to IP ranges and of course you can use IP tables and fail2ban too.