This is hardly surprising considering that most of the unauthorised access attempts your server will see will be automated bots trying port 22 and it is not worth while for the bot to try all other ports.