You can definitely self host. In terms of spam there's a bunch of config you need to do to not be put into the spam folder automatically.
I've done it for my side project, not saying it was easy and require a bit of linux knowledge but nothing you can't google.
Off the top of my head you have to setup your DNS and enable DNSSEC SPF and DKIM records Make sure your reverse dns is setup correctly DMARC as well
After that it's pretty much the same as your big email providers you build up your reputation by not sending out spammy shit emails then that's it.
Note: Be sure to secure your email server otherwise someone will try to hack it