I have been self-hosting my email for few years on Digital Ocean VPS. The biggest downside is that your messages are very often classified as spam by big players such as GMail or Microsoft, despite implementing DKIM, DMARC and SPF etc properly on my side. If it's important for me to get my message through, I have to use my "backup" GMail account, which pretty much defeants the point of self-hosting your email.