UUCP must stay; Fetchmail sucks (2001)
docs.freebsd.org
docs.freebsd.org
pop3 was never meant as a spool for MTA to MTA email transfers, but was instead meant for an MUA to access. It's not fetchmail's fault that the envelope TO isn't available via pop3. That, at the time, email service providers bolted on entire domain delivery to a single mailbox, and then users of those services put multiple recipients behind that single mailbox. Fetchmail made this work as best it could. Also missing at the time was any kind of serious connecting-client-as-MTA or MTA-to-MTA authentication (rather, relying on DNS and TCP!), which might have made async, receiver controlled MTA initiation more workable.
This was also due to common SMTP MTAs not having good (or at least easy/obvious to configure) store-for-later-pickup functionality where an MTA could query for email destined for itself rather than be always-on connected to the internet waiting for another MTA to initiate delivery. Fetchmail filled a hole that existed in email delivery at the time, but it was really a hole that should have never existed. The quality of the fetchmail code is another, albeit serious, issue in itself.
These days we have a better understanding of the differences between MTAs, MSAs, and MUAs.
I don't want to be "that guy" but we've known about this distinction since the 1980s. I know, because I was fringing in the discussions in standards ml during the x.400 years.
Many email clients, like Eudora and Thunderbird, were submitting email on port 25 for a long time, in the clear, where the only protection was very loose source IP address restrictions for outbound, otherwise you'd be an open relay. Using different servers for inbound vs outbound mail submission was a relatively rare thing too, because the routing can be a bear to set up. SMTPS (SMTP over SSL) on port 465 wasn't official and was locally used where it could be supported (by the client) and where an admin had bothered to configured it. Many email clients didn't have support for even using a non-standard port for any thing mail related, so even once the mail submission port 587 was standardized it was often a problem getting a client to use it or providing authentication on it.
But even the OP is saying that fetchmail is a problem because it's trying to be an a full, transparent MTA when it's really a glorified MUA. The fact that fetchmail (and its ilk) talk MUA protocols to receive email (pop3, imap) and can then turn around and reinject that message into the mail stream using SMTP to implement local delivery is where a lot of the problems with fetchmail come from (such as munging the headers or creating email loops with mailing lists).
Setting up a proper mta/mua/msa respecting service was hard, and for many people remains hard. Exploring how to get utf8 clean across all the m* protocols for lhs@rhs reminds me how complex this can be.
After nearly two decades, no matter what system I plunk it on or mail provider I use, the same stupid script just keeps working.
I used getmail for years and it is solid. duckerude posted this elsewhere in the thread, where the author of getmail rants a bit about fetchmail. http://pyropus.ca/software/getmail/faq.html#faq-about-why
Once I left Gmail I was interested in something faster and more efficient, because I was using the transfer tool for more than just backup, it became the main way I receive email on my machine. Then the mail gets indexed (for text search) by another program and displayed by a third.
Anyway I found mbsync, which is just this amazing, efficient, very solid transfer program that's been around for many years just quietly doing its job. I almost wrote it off because its homepage is on sourceforge but if you poke around a bit you'll see it is actively developed, there's an active mailing list, etc. I love it. It's kind of interesting how this project I had never heard of just does its job well and works and the program by the semi-famous software pundit whose programming essay went viral really does not (from what I have heard).
I haven't found a single other program that does this well; there are tools to sync entire directory trees that can idle, but they don't seem to support "download everything and delete it" the way fetchmail does.
Many e-mail services throttle new connections, so it's not the difference between getting your mail now and 1 minute from now, it's the difference between getting your mail now and 5-10 minutes from now.
[edit]
Another thing keeping me on fetchmail; I have all my filters setup with maildrop. Most other programs want to deliver to a local folder directly. Maybe there's a way around this?
For private email consolidation, I've been running fetchmail for probably 20+ years now without any hiccups or complaints whatsoever. I haven't had to touch my Fetchmail+Postfix+Maildrop setup in like 15+ years, except for the postfix part in the sense that nobody accepts emails from dialup IPs any longer, so everything I _send_ goes through the proper mail server, instead of my own.
My actual email provider (that I've also used for 20+ years, i.e. no, it's definitely not and never will be GMail) does throttle connections via POP3 or IMAP to 3 connections per 5 minutes. That's completely fine. I really don't care whether I get an email now or in 5 minutes from now. I check it way less often than that and I do definitely _not_ have notifications turned on for my email client. And when I do care (coz you know, some "2FA" BS email, I check via my provider's app or log into my server and start a `fetchmail`.
The bizarre thing is that no one online seems to have these issues? The mbsync configs for Gmail just seem to work for people, or something. I do have auto-expunge turned off in Gmail, so I’m kind of at my wit’s end with it. It’s a shame because it is an amazing program. It synced 10,000 emails without a hitch. I only started getting issues when I tried to sync from my maildir to the IMAP store.
Also, Gmail’s tags don’t map nicely onto folders.
There are also complaints about it being insecure and of low code quality, but those seem to rarely be the main complaint.
What has become of our beloved free(digital)land?
1. It's very easy to get to.
2. It has incredibly fast search that has 0 setup.
We have never really even tried to address problem 1 as an open source community. Networks, name lookup, and VPNs remain incredibly complex topics that beginners cannot hope to wrestle with. The best we have is .mdns which either works magically or perversely refuses to work.
Similarly for free text search, the software world simply hasn't delivered a lego-like solution for email search. You CAN rig up any number of open source projects but it is neither easy nor instant. And even other professional products like Apple Mail struggle with a mere gigabyte of email.
Despite the fact that it's 2021 and every successful email provider aggressively solves these problems, the open source world still debates about the utility of ubiquitous search or pretends that local networking isn't a pressing problem.
I kinda disagree. It's probably easier than ever to set up your own mail server, in some abstract sense. You can get a virtual machine, use docker, heck, someone can hand you a complete image that you just have to bring up and set up with some config.
The problem is, it literally doesn't matter how much the 'open source' community comes together, it simply can not provide a turn key solution as good as
Desired email account: [________]@gmail.com
Password: [________]
Verify Password: [________]
[X] I agree to have all my data used in arbitrary ways
Or at least, not in a sane way. I can set up a site where you feed me your credit card number and pick a domain name, and I set up your AWS account for you, register your DNS name for you, configure DNS, and stand up everything you need and set it all up... but then we've got a split ownership interest. I can hand it all back to you, but you don't understand the setup. I can give you root on the system, but when you change anything, my automation stops working.
I partially agree, but I think we could get a lot closer than we are now. It feels like the main reason this isn't possible is because you need to go through a registrar to get a DNS name, and that's tricky to do as part of a FOSS project. Maybe you could integrate with the APIs of a few registrars, but... it's not ideal.
As far as the "run thing on server" side of it goes, though, projects like Sandstorm[0] have gotten really far re: making it a simple process. I stood up this instance of Etherpad with a few clicks on a web UI, for example: https://sandstorm.terracrypt.net/shared/aR2HXaoLSkLuXLhhAQon...
Sandstorm in particular doesn't quite work for mail servers just because the software is heavily oriented towards webapps, but there's no reason a similar system couldn't work in principle.
I think something that could be like sandstorm but using docker containers might be a good thing, but there's still the "you need to hand me your credit card if you want me to register a domain for you" which is a big barrier vs. the screen I showed above. A credit card off the top is already 4-6 entry fields, and everyone knows to be a bit nervous about entering those....
I can't agree, getting a domain is much easier than it is to configure the abysmal shit that is Postfix+Dovecot+WebGUI. God forbid you want proper search as well.
What can't so easily be done programmatically is DNS registration, because it involves money changing hands and there's no standard "registrar API". You'd have to either support an arbitrary number of differently-shaped APIs of different registrars or pick a "blessed few" to support.
GoDaddy, GCP/google=domains, AWS, Azure, DigitalOcean and a couple others would provide enough for a LOT of people. Especially since godaddy and google domains include dns, and DO doesn't charge for it.
I've setup a script to make it easier to manage a handful of dev domains (commit to a repo, and it updates via build/event pipeline) with DO.. was surprisingly easy. I don't think doing similar for the biggest cloud/dns providers would be all that hard to integrate.
Spam/botnets have a lot to answer for.
Of course, it's not simple to set up a mail server that stays clean of spam RBLs, or that is Gmail-acceptable out the gate. But that's just the bar that got set as people went to Gmail because of the lack of other good alternatives.
Question is, is Gmail hegemony fixable now?
1. End to end encryption 2. search as a first-class feature 3. decentralized/federated 4. standard minimal rich text format (maybe something like markdown), instead of an inconsistent subset of html 5. Fix some of the legacy limitations of email, like having to be ascii safe, line limit of 70 characters etc. 6. Possibly make it easier to have an identity that isn't tied to your service provider
The big questions are how would you get the general population to switch from email, and what to do about spam.
It supports many things that markdown does not (setting font families, sizes, and color for one example), and nobody uses it anymore.
2> Your MUA does that. Get one that doesn't suck.
3> Impossible and/or dangerous.
4> No. HTML is NOT part of a MIME standard, even.
5> That's an issue of netiquette, often you have to post lines over 70 chars in News servers, too.
6> That's a different issue.
2. I have tried several MUAs and haven't found one that IMHO "doesn't suck." Although search isn't always the problem. (The biggest problem I've run into is actually getting them to work with logging in using an SSO method like SAML or OIDC). And this point was specifically in response to the parents statment that "Had IMAP addressed search in a better way, I think it would have made a huge difference."
3. Not sure what you mean by that since email _is_ decentralized and federated. Meaning, if you use gmail as your email provider you can still send messages to people using a different email provider. As opposed to say something like Signal or Whatsapp, where you can only message people using the same service as you.
4. Well, some subset of html has become the de facto standard for rich text in emails. Although what subset that is depends on the client...
6. Different issue from what? It is a very real complaint I have with email. Changing your email address is a pretty big pain, possibly more so than changing your physical address or phone number. For many people, myself included, the thought of having to change my email address is a pretty good motivation for staying with the service I currently use. Now it is possible to use your own domain with many email providers. But doing so is beyond the technical skills of most people, and costs more for essentially insurance in case you stop liking your current email provider some time in the future. Unfortunately, I don't know of a good solution for this. At least not one that is scalable.
Maybe, if we get maddy working near-perfectly and kept up-to-date.
3. dealing with spam is a massive PITA
Seriously, mu4e search on modern hw is incredibly fast (and I have a 10 year work email archive).
Thunderbird plus global search
Recoll + client of choice
I used to run personal email server some years ago. The spam was harder and harder to manage - first local SpamAssasin was fine, the I had to tweak rules, then I tried distributed filtering (DCC)... and then I gave up and started paying for Fastmail. This ended up working much better than I could ever do.
Usenet was similar -- I remember looking at the newgroups which were at more than half spam (and some were 95%+ spam) and then just leaving for forums. (plus there were a major UX annoyance: unlike with email, there is no simple way to sync read status across multiple devices)
Our free(digital) land has been killed by a handful of bad actors. I no longer believe in "free for all" systems. The best we can do is many independent systems, each with its own rules and policies.
In ten years of running this mail server, I have only had one persistent spammer I couldn’t manage to get filtered with these settings, but then I just created a custom rule.
And you're right, the centralization is crazy, and opposite of what techno utopians predicted/promised at the time.
There was no ISP in my area back then, not even dial-up. Again, around 1991.
Isn't that exactly what Envelope-to is supposed to be? Protocol-level data? It's not really part of the e-mail, but added when sending it. If you forward the mail to another machine, there will be a new envelope.
I can see that it's all a gray-ish area with email (because email is a mess (which might be part of why it's been successful)) but I can't see a reason to complain about losing Envelope-to.
If you don't want to do that, at least read the wikipedia page: https://en.wikipedia.org/wiki/The_Cathedral_and_the_Bazaar
It's also responsible for hatching one of the great canards of open source, "Linus's Law" (I'm sure Linus is thrilled with having his name attached to that), "many eyes make all bugs shallow", which has in the last 10 years become one of the great jokes of computer science.
He wasn't completely wrong about everything, but the main premise - that careful planning and development yields worse results than guerilla ("bazaar") way of working - didn't pass the test of time. His main argument was the Linux kernel. But when you look at this project now, it seems quite far from the bazaar style. There are many bazaar-style projects that failed and many cathedral-style projects that do very well. So his main argument is moot.
We can treat this essay as a historical document, not something anyone should refer to as to how the open source development should be done (even though he has some good points about leadership style for example).
It's why I'm rereading this book today.
One thing I've been thinking about since morning was ESR's "Linus Law", "given enough eyeballs, all bugs are shallow". Today the idea is heavily criticized. The very existence of fetchmail by the author was a counterpoint - the code quality was below average, it had numerous security vulnerabilities, and arguably basing the whole book on fetchmail is the book's greatest flaw (not to say the criticisms of ESR's personal qualification from many hackers).
But if you read the original context...
> Linus was directly aiming to maximize the number of person-hours thrown at debugging and development, even at the possible cost of instability in the code and user-base burnout if any serious bug proved intractable. Linus was behaving as though he believed something like this: Given a large enough beta-tester and co-developer base, almost every problem will be characterized quickly and the fix obvious to someone.
The original context was strictly about beta-testing - about how the Linux kernel didn't collapse on its own weight despite the lack of a BSD-style "core" team and heavily scrutinized release. Instead, it achieved system stability with the help from a large number of beta-testers and co-developers who are willing to run on the bleeding edge. It's still how the kernel and the majority of the FOSS world. "Build today's git snapshot and see whether it explodes" can be quite effective.
Then ESR simplified it to...
> Or, less formally, "Given enough eyeballs, all bugs are shallow."
And now it means "people can find all bugs, because there are always a lot of eyeballs to read the source code". Now it can refer to all sorts of things beyond its scope. It is a serious mistake. Notably, security vulnerabilities are different - it usually does not have any user-visible effects and can only be triggered by carefully crafted malicious inputs, not normal uses. This is the reason it doesn't work for security. Thus, maybe a more accurate reflection of this rule could be: Given enough eyeballs, all user-visible bugs (e.g. crashes) are shallow.
At the end of the day, my conclusion was: Perhaps "Linus Law" was not as wrong as it seemed. ESR simply misinterpreted his observations.
Admittedly, this was 19 years ago, and it was a relatively small fix.
Of course, fetchmail wouldn't have the envelope info, because that occurred in an SMTP chat that is not part of the message landing in your inbox.
Something that accesses delivered mails in IMAP or POP3 boxes cannot be a full-blown MTA. E.e. we can't program a mail exchanger machine to accept the mails on behalf of hundreds of users, by delivering them into a giant mailbox, and then continuing the delivery path by sucking them out with a fetchmail-like tool, and forwarding to more forwarding hosts.
The whole complaint of this irate BSD person is that you can't do this kind of nonsensical thing.
There is a sort of valid criticism in that fetchmail (or at least did originally, I don't know about now) worked by re-sending the messages to the user over a mail transport mechanism, rather than delivering it into a mail directory structure. E-mail clients that download and cache messages certainly do not do anything of that sort. However, that shields fetchmail from having to know how the user's mail software likes to have the messages stored. Arguably, sending the mail is the right interface.
Though the envelope info isn't available, it's assumed that it was already processed by the remote mail server. The remote server did all he spam checks on the SMTP info, and whatnot, decided that the message is deliverable to the user, so there it is.
Fetchmail assumes that if the mail is in the remote box, then it's yours, and can be effectively forwarded. The envelope sender is Fetchmail itself.
If that is broken, so are mailing lists. When you receive a message from someone via a mailing list, you also don't have the envelope which the mailing list robot's server received. The envelope is that of the mailing list's own delivery; the mailing list robot is the MAIL FROM: sender.
For that matter, you are not even the original intended recipient! The mail was originally to the list, and now it is to you.
Not having an envelope isn't a show-stopping impediment to further forwarding and delivery. You just have to understand and accept that it's more of a forwarding style of delivery, like a mailing list robot.
Overall, it's almost like complaining that a dog which fetches the mail from your door and brings it to the kitchen is not a trained postal worker, does not understand and handle addresses right, and could not be reliably used to service an apartment building with six tenants, all by itself, such that the mailman could just put everything in one bin and leave it to the dog.