Well sure it could be bad if you are using 10 computers or you have 10 different servers to login to, but personally I have 2 computers that I use and only manage a few servers so it isn't all that bad, plus the config file is linked to dropbox for me.

Besides it is more like a one time annoyance unless you are one of those crazy people that formats without backups every month.

Yes, this is true - it's not going to stop someone who really wants in from finding your SSH port. But at the very least, it's useful when looking through your logs, because the ONLY failed log in attempts you will see are from people who mean business.