https://www.w3.org/Daemon/User/Installation/PrivilegedPorts....
https://adayinthelifeof.nl/2012/03/12/why-putting-ssh-on-ano...
https://www.w3.org/Daemon/User/Installation/PrivilegedPorts....
https://adayinthelifeof.nl/2012/03/12/why-putting-ssh-on-ano...
Privileged ports is a measure to guarantee that daemon listening there was started with root privileges. Which is a useful guarantee on a shared server, less so on a single-purpose or personal server.
Plus, if you have an sshd started at boot, and already listening on a pre-defined high port, there is very little chance another program could bind to the same port, save crashing and racing the daemon.
Finally, in ssh protocol, server authenticates itself to the client, so even if by some trickery a bogus ssh server would bind to the same port, it will either need to access /etc/ssh/ssh_host_*_key, which are enforced to be root-readable only, or risk being trivially detected by a client comparing presented public key to that saved from last login in the ~/.ssh/known_hosts.
There are also some DPI firewalls, or just port-based firewall that can block ssh traffic on a non-standard port, but assuming we are talking about your own server, that should be easy to rectify too.
There's no harm in having connection attempts on your ssh port. Changing your port will not make you more secure against a dedicated attacker, and you just make things harder for yourself (and marginally less secure, for the reasons you mentioned) with no real benefit.
And like I mentioned in the other comment, just use SSH keys and disable password auth. It's not hard, and it's all it takes to thwart every bot that hits you on port 22. If you want to be more hardcore you can run fail2ban or firewall the port down to your country or whatever.
It's the equivalent of "I connect to my host via IP address instead of having DNS records because it'll be harder for them to find!".
You still paint camoflage on a tank.