It takes about 8 minutes to scan the whole internet for port 22. If there's an SSH 0day, and I have a botnet, I am going to make the entire botnet scan the entire internet on every port, because I know people move their SSH port. With 5,000 hosts, I can scan all 65,535 TCP ports across the entire internet in 104 minutes.
How fast can you patch your server after the 0day's been announced? What if you're asleep?
The time it takes to change the SSH port could also be used to harden the sshd config, which would eliminate several classes of previous exploits. A little more time and the entire system could be hardened, so that even if you are attacked, the attacker can't do much of anything.