Usability studies shouldn't just be focused on the UI but equally on the setup an maintenance of server software. You would need to find a balance though since you won't want to remove features just because of ease of setup.
Sorry for ranting a bit and take the discussion a bit off topic.
Software is complex but it would be nice if us developers put energy into making complex setups feel less complex. It might even benefit security and less malconfigured services.
Sounds like you're describing OpenBSD. With the slight catch that they're usually not making complex setups just feel less complex, but actually be less complex.
It's the OS where you're expected not to twist any knobs unnecessarily (and they're actively looking for ways to eliminate knobs that don't help); the defaults are meant to be safe and sane, so that you don't need to become a master sysadmin overnight.
Of course it's not for everyone, when so many people seem to want complex setups and loads of features. They also want to twist all the knobs.
There are hundreds of thousands specific needs for a company, a small group or even a single person.
As soon as you know what you need, you will soon find a setup advice.
Also, if you don't know what is possible via email protocols (smtp, pop, imap etc.) you are thinking about solutions that has been thought about several times and mostly you start to "hack" something (via Perl, Python, Milter etc.) to fit your visionary idea, just to realize that it is a common problem that you can switch on and off via config.
Or you can do a simple setup with fewer than ten lines of configuration for the mail daemon (written in a matter of minutes) with no extra goop, on a system where the tools are integrated so you don't need to figure out how to install anything.
In a fully trusted and firewalled network, installing a mail server is probably as simple as `sudo apt-get install <mail server>` and accepting whatever debconf prompts you with.
But if you want an Internet-connected system and you want to send mail to users on outlook.com and gmail.com, then it's a different beast entirely, for one simple reason: spam.
Two concerns: incoming spam, and outgoing spam. Dealing with spam is an arms race; in the 90s, a simple bayesian filter on the contents of the email (such as POPFile) would be sufficient for most users. Spammers have gotten more sophisticated, and so tuning the rules so that you're not inundated with spam takes quite a bit of time.
On the flip side, other people have pretty sophisticated email filters that look at things like email signatures (DKIM), DNS records (reverse DNS, SPF), whether you're running a server off your home ISP connection (DNSBL), and all sorts of other shenanigans. Configuring a mail server to pass all these checks is supposed to be hard enough that you'd only want to do it once -- if it wasn't, then the spammers would be able to do it quickly and users would be getting a lot more spam because every time you block one server, the spammers would just set up another one.
Also, depending on the mailserver selected, installation difficulty varies. And it can be harder your first time if you're setting up SSL. Took me several weeks to understand Apache HTTP server's reverse proxy set up and figure out how to modify it to support forward secrecy.
I've found Zimbra to be easy to install and a delight to use. There's lot of great features such as spam filter and calendar. It's open source.
Don't forget to set up your PTR (reverse DNS) which often involves specifically asking cloud providers to set this for you. Otherwise your emails will almost always be tagged as spam.
The best thing to do is to find a solution that pretty much sets itself up and don't tweak the knobs unless you need to. Probably someone on here can suggest a nice solution.
Also, being a smart programmer has nothing to do with skill in setting up servers, which is more of an Ops task. It's one thing to write good code, it's a completely different skill to walk through installation manuals and editing config files.
Specifically, you can find links to https://github.com/al3x/sovereign and https://github.com/darklajid/ansible-communication which are of the DIY type. Other projects are mentioned as well that are sponsored projects or WIPs.