You can just set it in your SSH config for that destination and it'll use that port every time you do ssh destination.com.