Really using port 22 is very ill advised anyway because you will get constant nuisance brute force attacks (accomplishing nothing because you're using keys or certificates I hope) but still eating up cycles for the crypto handshake.
A lot of servers expose something public so they can be found. Otherwise what's the point of being publicly accessible?
1. You listen on IPv4 and someone probes all the IPv4 space and your server announces "Hi, I am web123.example.com" or similar in its responsible
2. You have HTTPS on the server and the HTTPS address ends up in the certificate transparency logs.
3. You have a public service on that server and announce the address somewhere.
But when you have billions of IP addresses, why does SSH need to listen on the same address as HTTPS or anything you're running publicly? It's also infeasible to probe the entirety of IPv6 space the way you can probe all of IPv4, even though we're only assigning addresses in 3/65535 of it right now.
I mean, yes, you'll get a constant stream of them on IPv4, but why would you run a server on v4 unless you absolutely needed to? The address space is so small you can scan every IP in 5 minutes per port, and if you have my v4 address you can enumerate every single server I'm running just by scanning 65k ports.
Meanwhile, on v6, even the latter of those takes a thousand years. How would people even find the server?