A long time ago I used to have a general purpose server on a DSL line with port 22 open. I thought everything was fairly secure, patched up, etc... I never thought brute forcers would be able to guess one of my accounts (which was a friend's account actually). The kicker? The password was the username. I found out about the rootkit a few days after they got in. I tell ya, kids these days just don't know how to hide a backdoor.
A really good port knocker is your best defense against rogue attackers of an external service (because there is no defense against a 0-day). For an iptables-supported system I recommend Knockknock by Moxie Marlinspike (http://www.thoughtcrime.org/software/knockknock/).