Privileged ports is a measure to guarantee that daemon listening there was started with root privileges. Which is a useful guarantee on a shared server, less so on a single-purpose or personal server.
Plus, if you have an sshd started at boot, and already listening on a pre-defined high port, there is very little chance another program could bind to the same port, save crashing and racing the daemon.
Finally, in ssh protocol, server authenticates itself to the client, so even if by some trickery a bogus ssh server would bind to the same port, it will either need to access /etc/ssh/ssh_host_*_key, which are enforced to be root-readable only, or risk being trivially detected by a client comparing presented public key to that saved from last login in the ~/.ssh/known_hosts.
There are also some DPI firewalls, or just port-based firewall that can block ssh traffic on a non-standard port, but assuming we are talking about your own server, that should be easy to rectify too.