All of the same arguments can be made for SMTP, and hosting your own server there.
In the fediverse calls to defederate happen pretty regularly, and the reasoning is sound—your instance will be publicly mirroring any content on a federated instance, so if you don't want to publicly host extremist views (from either wing) your only option is to defederate from instances that allow them.