This is fine, though. "Security through obscurity is not security" but moving your SSH port to something not 22 will utterly eliminate brute force attacks.
It's too much bother to go find it, and the bozos will just move on to the next machine with port 22 open.