Ask HN: Is is safe to change my domain's MX records away from Google?

I use Google to manage mails both for myself (personal site) and that of my company. I just set the MX records of these domains to Google's mail servers and it's been working fine ever since.

However, with increasing frequency of the dreaded and un-recoverable Google bans, I am wondering if changing the MX records of a domain to a different email provider has any complications associated with it.

For eg. will an overnight change in MX records be flagged by prominent email providers, resulting in subsequent mails sent from that domain consigned to spam by these providers?

While I know that existing email data may forever be locked out from me, will at least all subsequent communication with other services I use (AWS, Github etc.) or even communication with official Govt agencies (tax department etc.) work fine without getting blocked by spam filters.

My apprehension stems from my understanding that email server management is hard to get working right especially because of how other mail servers flag mails from your server as spam unless it's configured just right. I admit that I'm not very knowledgeable about it. I'm hoping more erudite people here can share pointers/warnings/tips.