Get a range of ip addresses where you will log in from (your DSL address ranges from home for example), and block all other ip addresses on that port. You may want to add some other hosts you control to that whitelist as well as a backup in case your ISP changes addresses. If you are the only one that is supposed to access some ports, then block everyone else. (use last -a to see all the places you've logged in recently to start making your whitelist).

You can also rate limit new connections to ssh depending on your use cases. So even if your whitelist is beaten you limit the number of attempts.

Now add fail2ban, so that even at 10 new rate limited connections per minute they get banned after X failed attempts.

Change the default port of ssh, to make it harder to find.

Remove password authentication, and make sure you have good secure keys.

You can also disable sudo, and root logins perhaps. As well you could try 2 factor auth, and perhaps move your keys to a secure(encrypted) USB drive.

But really, whitelisting who can connect to ssh in the first place will stop 99% of the automated brute force attacks, with all the rest stopping the remaining 0.99%.