So you are accessing that server's services from some network, why are you not only allowing SSH over that network?

Or, if your service is open to the internet then why does not what I said above hold true?