Those scans/bruteforce attempts are mostly harmless anyways, so there is very little benefit in changing the port. Has there been yet a single drive-by 0day attack against opensshd, because that's essentially the only thing the port change would help against.