Having a hardened email server and good passwords does nothing for your security if the attacker targets and takes over either:
* Your DNS Records - MX records point to your email server, the attacker just changes the record to point to their email server, taking your hardened email server out of the loop.
* Your Domain Registrar - the attacker can just change your DNS records to point to their own DNS servers (with their own MX records pointing to their own email server).
Do you trust your DNS provider is completely secure from hacking and social engineering?
Do you trust your Domain registrar to adequately secure their systems from hacking and social engineering? Remember, you only pay them ~$10/year for your domain.