sshd is hidden behind port knocking, for what that's worth. (No, I do not want to have a religious argument about port knocking this morning)
The real key to the simplicity and security of my setup is email over SSH with a console client. Not only does that remove oceans of attack surface, but it also means you can read email without the emails themselves traversing the network. I guess the characters go over SSH, but that's not quite the same thing...