I think it's just a matter of numbers. The IP search space is already pretty large with IPv4 on just a single port. With IPv6 it'll be more or less impossible for attackers to make any substantial progress by just searching for IP addresses to attack, let alone an IPv6 address on a non-standard port.
After reading this I'm likely going to be upgrading my technique. SSH on IPv6 only (eventually shutting down IPv4 altogether, can't wait), listening on a non-standard port, limit 1 connection per IP.