You set yourself up to miss a fair bit of email that way. Most mail servers I work with use "opportunistic TLS": attempt TLS for outgoing mail; accept if offered for incoming mail; selectively enforce TLS on a per-domain basis for confidential communication to partner banks, law firms, financial firms.

Enforce TLS on your mail server and you're siloing yourself from a large part of the internet, and unless you're Gmail and your actions carry weight, others will simply stop emailing you after their mail keeps bouncing.