Thanks for the reply. Fastmail is indeed nice (I've used them before but not hosting my own domain). But at this point, I'm not willing to risk moving to another player besides MS or Google.
After 2 occurrences of my mail goes into spam folder, I became paranoid and always BCC my gmail account for every important email.
In the end I say "what the hell" and subscribed to O365. Google Workspace is too expensive for me.