It would be interesting to know exactly how one could get owned by running just an email server. It's been forever since there has been any server that could be attacked with just SMTP. IMAP has a login, as does that buggy PHP webmail program.
There just isn't much of an attack surface there.
The real problems are spam and convincing other mail servers to accept your email.