So to get around this you either have to allow SSH from anywhere or you have to use some remote KVM system. Most of the remote KVM systems seem to be based on Java applets which is not really something you want to enable on your system.
So what is the best way to get around this? Just open up SSH and be diligent about your SSH security? implement port knocking?