low security impact, best practice as others has mentioned
If you strictly use public key authentication and don’t expose your SSH server to the public internet, this is a non-issue. Furthermore there’s no defensible reason not to do that - it’s very straightforward to firewall SSH access behind a private network, and VPNs exist precisely for this reason. Access to the network and access to machines within the network should be strictly decomposed. If you allow SSH access from the public internet and have a VPN, you have to worry about two (privileged) points of failure, not one. Put everything behind a public-key authenticated VPN and SSH vulnerabilities are obviated, with the exception of insider threats in the organization.