There have been remote code execution exploits in mail servers before, Exim most recently. There probably are more hiding somewhere in one of the many layers that are needed for modern email servers.
Since a server is a much juicier and visible target than any client, expect attacks.
Why do you think that?
Isn't a web browser that has access to your gmail inbox just as juicy as your self-hosted imap server?
I don't really see why a "server" would somehow be inherently a more interesting target. Google's servers, sure, they give you acess to billions of mailboxes, but your own private email server?