Email Explained from First Principles | Hacker News Reader

Email Explained from First Principles

explained-from-first-principles.com

106 pointsthunderbong35 comments

Back in the early 2000's I worked for a company that was the lead firm doing analysis of email spammers for later targeting by the law firms of the big free email providers.

The phrase "be liberal in what you accept, and conservative in what you send" certainly applied to SMTP which led to all kinds of craziness in spam.

For example:

- Emails with multiple "From" fields in the headers

- Totally bogus relay chain data

- all kinds of other garbage in random fields

- sender IPs that were faked etc

This was also around the time that spammers were switching to using bas64 MIME encoded images to get around text filters.

The company I worked for actually got pretty good at analysing the spam thanks to a combination of a IMDB style web based browser for the spam I put together and good old fashioned "leg work" to track the owner of URLs, merchant accounts etc. That in turn led to some of the spammers actually shutting down.

More details here: https://twitter.com/alexpotato/status/1208948480867127296

>> Emails with multiple "From" fields in the headers

Do you think this is still possible?

Sure better be, for the sake of backwards compatibility.

This isn't a problem for mail transport as the recipient is specified in the envelope and the message itself is not examined.

What a client should do in such a case is unclear. I suspect that they all use the first one encountered and ignore the rest.

Multiple From: fields are allowed by https://tools.ietf.org/html/rfc5322#section-3.6.2 as long as a Sender: is also included.

Just to be precise: It's a single From field with several addresses, not multiple From fields (see https://tools.ietf.org/html/rfc5322#page-21).

RFC 6854 updated the From field to allow an empty group construct (see https://explained-from-first-principles.com/email/#no-reply) but mainly for other reasons (see https://explained-from-first-principles.com/email/#email-add...).

DMARC authentication either rejects multiple addresses in the From field or authenticates them all and applies the most strict policy (see https://explained-from-first-principles.com/email/#domain-ba...).

> - Totally bogus relay chain data

- all kinds of other garbage in random fields

- sender IPs that were faked etc

Anyone interested in looking at all the possible permutations of spam, can view the spamassassin rule sets which are quite comprehensive by now. Spamassassin has been around for close to 20 years.

I don't know why, but the fact this isn't in the format of a blog makes me wary. It gives me "aggregator" vibes. However, it sure seems like a ton of dedication went into this writeup. I'd love to hear from somebody with domain knowledge who has read some of it if it seems good (i.e. factual/complete and well explained, subjectively).

Edit: Oh, it is a blog. Something about it made me assume it was like a WikiHow.

Thanks for the feedback. I summarized over a 100 RFCs in this article so I certainly hope it's factual and complete. :-)

I think what happened is I saw the breadth of the writeup, saw the title of the website, and thought "whoa, there are other writeups like this on other topics? It must be a platform anybody can submit to." That's a complement - this is an impressive undertaking.

I love the style so far. It hits that sweet spot where it's thorough, like documentation, but isn't obtuse, instead being written in prose, like a typical blog post, but without being wordy and overly conversational. This is truly valuable.

This is like the Encyclopedia Britannica entry for 'email'. I wouldn't call it "from first principles", I'd call it "a survey of everything related to email and email delivery" (propane and propane accessories).

Great timing for me. I've spent the past week setting up a new "Mail-in-a-Box" email server after procrastinating past the point of upgrading my old one.

I'd much rather not run my own email server but after 20 years of developing web apps that need to send emails I've concluded the only thing worse than having to build and manage my own is using 3rd parties to handle email.

That said, the latest MAIB is pretty sweet and it's a bit easier to get de-blacklisted now than it was 5-10 years ago, but it's still a pita.

Why is using 3rd party email providers to send email so painful?

The succinct reason is you're dependent upon them.

Over the years I've invested time in learning and implementing their API only to see them get acquired by a competitor that told me I had to use theirs. And early on I had one spamming with my domain names. And I had one shut down without notice.

Nowadays they can be expensive. You can spin up a $12 a month DO server that will handle quite a bit of mail.

For me, MAIB having a DNS server built-in was a big plus too. I have several domain names and email being sent by them or just email addresses I used for others to contact me, and alias addresses that all go to one email box, and some of my domain names are just for fun. For those I can create a quick email alias so it sends email to an existing account.

I use my DNS server IP address first on the list of dns servers in my network preferences on my Mac so when I make any changes to the sites I'm working on they show up instantly. I don't have to wait for them to propagate to find out I broke something.

So, it's really more than just email, but having that completely under my control has been a good thing. A pita sometimes, but still a good thing.

What VPS provider are you using? I set up MiaB on Digitalocean a few weeks ago only to find out they won't open port 25.

DigitalOcean.

They have entire ranges of IP addresses block by some of the blocklist but my users are here in the U.S. and from what I could find those are mostly used outside the U.S.

It's a hit or miss thing. If you spin up a new "Droplet" you should do an IP blacklist check on before your build your server. If it's on more than just a few you might want to start over.

This article is sort of a mess and I find it difficult to imagine anyone could learn about email from reading it. Too often it starts with something misleading and only clarifies later. For example, at first is says that recipients are given by headers in the message, but of course this is not the case. Recipients are part of the envelope. Envelope recipients are sometimes derived from the message, but sometimes not. I think it would be better to explain the SMTP envelope recipients first, which would lay the groundwork to explain that sometimes the recipients might be derived from RFC-822 header fields created by an MUA. But it's important to understand this might not always be true (there might not have been an MUA involved, or a webmail system with integrated mailer, like Gmail, might not need to put blind recipients in the message).

Well I can still probably learn a lot from it.

Truth is if you're an app maker it's a pretty deep dive into just setting up an email server and getting them de-blacklisted. It's easier than it was 10 years ago but it's still not really "simple" or "easy".

MAIB is the best I've found for my purposes and much better for those than the other options they mention on their website or anything else I could find.

If you've found flaws in it you might reach out to the author and point them out. If it were me I'd appreciate that and make edits and give credits in the footnotes to those who took the time to help me.

Yes, I'm always happy to receive feedback, either via email or a GitHub issue: https://github.com/KasparEtter/ef1p/issues :-)

Thank you for your feedback. I see how this can be confusing. The first chapter is explicitly about user-facing concepts, though. I explain message envelopes here: https://explained-from-first-principles.com/email/#message-v...

Right afterwards, you find an example of how and why the envelope diverges from the message header fields, followed by an in detail treatment of who removes the Bcc header field, and how Gmail recovers the Bcc recipients.

But can it explain how a financial institution can reliably send me a 2FA passcode in a few seconds via email, but a message from wife in the other room can take a couple of hours to arrive?

Presumably because the financial institution sends the email directly to your email provider via SMTP while your wife sends the email to her email provider via a mail client protocol like POP3 or IMAP, and then her email provider forwards the email on to your email provider via SMTP at its leisure

Does anyone actually implement the BURL extension to IMAP? I think pretty much all mail clients just use SMTP for submission.

Not a bad guess, except that gmail is the email provider for both my wife and myself.

Then the answer might be that the bank does not use Gmail. It is after all known to be not that great.

Gmail doesn’t send instantly. This is why they could add the undo feature for sending emails.

That explains a ten second delay, not a couple of hours as parent poster claims.

Anybody else get a save password prompt for localhost:password when accessing this site?

Yeah, I'm sorry about this. I realized this only once the post was live. I will see that I can fix this. Did you access the page with Firefox? And don't worry, it's nothing malicious. It's just the ESMTP tool, which has a password field: https://explained-from-first-principles.com/email/tools/#esm...

Yes, it was Firefox. No worries, it was just odd.

I love this. I noticed there’s also an article in the same style explaining the internet. Looks like I have my reading material while I wait for my vaccine today!

Link?

https://explained-from-first-principles.com/internet/

Oh, look mommy, networking explained without mentioning OSI model! Good stuff.

Sorry, not sure if you’re being sincere or not!

No sarcasm. No sarcasm marker.

Reply on news.ycombinator.com