Ask HN: Is it hard to host your own podcast?

If I understand correctly It's just RSS Feed correct? Then I feel I should be able to host one easily on my own website. Are there any gotchas that I'm missing. Like hosting an email server?

Email servers has problems like

- getting a good ranking ip

- filtering out spam

- managing security updates

etc