I connect to many SSH servers that aren't under my control. It's pretty annoying to remember an arbitrary SSH port for my own.
Binding to IPv6-only is more effective at reducing log spam: IP scanning 2^128 addresses is impractical, and scanners often cannot connect because of misconfiguration/incompatibility or lack of a routable IPv6 host address.