Hacker News Reader

Comment by rolleiflex | Hacker News Reader

Great work, and as somebody who self hosted my own email from 2013 to 2021, I don’t envy you. What broke me down was Google starting to spam my emails that were replies to conversations that I did not start — even with a stellar domain reputation and DKIM, SPF, reverse DNS, greylisting, everything set up right.

I hope you have personal contacts on the Gmail team at Google, much as I’d like for this to be a joke.

View on HN

I had this exact same problem. When I finally got hold of someone who worked for Google (via a friend of a friend on a mailing list) and with the ability to check whatever their logs were claiming, I was informed that my domain's reputation was "too recent" or something like that.

My domain is a year older than Google itself and has been in continuous use for e-mail that whole time. The IP addresses it is on haven't changed in a decade. But that didn't matter. DKIM, SPF, DMARC, forward and reverse matching DNS, exactly four users who do not send spam under penalty of being buried under legal solicitations for green cards, and all the rest didn't help. Randomly getting sent to /dev/null for no good reason. And not enough traffic to qualify me to use their Postmaster utilities.

Three years ago I gave up and ported to Fastmail with a tear in my eye for the days when even the smallest net on the Internet could be a full participant.

Same here - I HAD to move over to Fastmail because like GP I'd respond to email enquiries from GMail addresses and then never hear back. After weeks of this, someone finally phoned up to complain about my poor customer service. This despite also having an old and apparently well set-up domain.

I lost a lot of business back then. Thanks, Google.

I have a legacy GSuite account. Google filters email from Facebook and Microsoft sometimes. Hell, I think I might have even seen them filter their own mail once.

It’s a clown show. The opaque filters are just an excuse to engage in anti-competitive behavior IMO.

To be fair Facebook used to send an enormous amount of spam emails, at least to me. Years ago when I used it somewhat regularly they used to constantly shuffle their contact preferences and the effect was that I was constantly opted back in to an insane amount of emails. Eventually I just started marking them as spam until Gmail blocked everything they send. I've seen family Email addresses fairly recently full of Facebook junk too so I assume they still default to sending a lot.

I really enjoyed the whole process of running my own mail setups, configuring Postfix or OpenSMTPd to do clever aliases and transports, setting up CARP failover relays, managing DNS records, experimenting with different SPAM-filtering techniques, SSL, IMAP, mailing lists, the works. It taught me a lot about networking and security and everything a good sysadmin should know.

I find it sad that that‘s pretty much a waste of time now for most use cases.

I ended up doing the same for the exact same reason. And even when using fastmail, my email still sometimes gets blocked.

FTR, directed at those who valiantly continue to self-host mail/SMTP: Greylisting is not sound any more in this day and age, because the largest mail services will rarely, if ever, use the same MTA instance to retry delivery upon a soft bounce.

The good news is that if you have postfix, using postscreen with an informed choice of blocklists is enough to deal with 99%+ of inbound spam. You can strap in rspamd or spamassassin/amavis behind that, but it's mostly not needed.

The inbound-mail-problems are largely solved, but surefire delivery to other parties is a matter of IPaddr/domain reputation, properly implementing relevant standards, and luck.

If you're interested in learning more about (including, but not limited to, self-hosting) email, the #email channel on the libera.chat IRC network is a great resource to ask questions.

> FTR, directed at those who valiantly continue to self-host mail/SMTP: Greylisting is not sound any more in this day and age, because the largest mail services will rarely, if ever, use the same MTA instance to retry delivery upon a soft bounce.

I had this issue with SendGrid years ago: long story short, after discussing with support and eventually an engineer it turns out they weren't just looking at the status codes, but at the status messages. I don't recall the exact patterns they used, but they will retry if the message matches a fixed set patterns, and otherwise it would just discard it.

There was some back-and-forth over this, because our customer just had greylisting with the "wrong" error message. To be fair, they did turn it off for a few hours and took the conversation serious (none of this "we have passed it along", never to be heard from again) but they got back to us they turned it back on again "because the queues got too large". I mean .... okay.... Seems rather curious to break a fundamental aspect of email because "muh queues". Not having to worry about this sort of stuff is exactly why we're using SendGrid in the first place :-/

My experiences with MailGun were also not exactly stellar. At the time at least, these people literally did not understand how encodings worked and would mangle e.g. Greek or Hebrew emails in ISO-8869-7 or -8. Why? Well, turns out that "emails should be in ASCII or UTF-8 and there is no way for us to know which encoding is used". Ehh ... there is literally a header telling you... I sent a nice detailed email explaining this: no reply. Some follow-ups over the course of a few weeks: no reply. A not-so-nice snarky email inquiring whether the entire MailGun team was suffering from a horrible debilitating disease and if there was anything I could do to help: "well, we just didn't know what else to do as there is no way to solve this"...

I'm hardly an "email purist"; I understand there are practical concerns and the RFC isn't a stone tablet from the mountain. But this was just ridiculous. There are a bunch of other cases both SendGrid and MailGun are actually quite bad at.

Dealing with email providers is always a frustrating experience.

I haven't used DCC, but it looks like an interesting anti-spam toolkit:

https://www.dcc-servers.net/dcc/

It appears to support "weak" IP matching:

> All or part of the IP address of the SMTP client can be optionally ignored by DCC clients as far as the greylist triple is concerned. This feature may be useful for legitimate mail systems that shuffle messages among SMTP clients between retransmissions. See the dccm and dccd man pages.

https://www.dcc-servers.net/dcc/greylist.shtml

It doesn't quite sound like it does a job of 100% "same email, same sender, different mx in same domain" -but I suspect it works well enough in practice?

> Usually the DCC greylist system requires that an almost identical copy of the message be retransmitted during the embargo. If weak-body is present, any message with the same triple of sender IP address, sender mail address, and target mail address ends the em-bargo, even if the body of the message differs.

> If weak-IP is present, all mail from an SMTP client at an IP address is accept after any message from the same IP address has been ac-cepted.

https://www.dcc-servers.net/dcc/dcc-tree/dccd.html#OPTION-G

I can't recall what I used for greylisting last -possibly greylistd.

Anyway, the smart play these days might be to whitelist/greylist via SPF - I'm not sure if spammers (of the variant caught by greylists) generally have SPF?

See: https://poolp.org/posts/2019-12-01/spf-aware-greylisting-and...

https://github.com/poolpOrg/filter-greylist

Ed: although if service providers like mailgun simply ignore rfc and only "sometimes" retries... wwell that's a problem.

I actually haven't had huge problems with Google (yet). I'm not sure why.

My experience has been cyclical, as long as you stick to IPv4 with good SPF and your reverse DNS works it will work 95% of the time. The remaining 5% appears to be completely random however.

It’s possible that my issue was too low a traffic to ‘hold onto’ my score with Gmail, since it was one domain and one email address. With some luck you should be able to have enough traffic to avoid that. Best of luck.

Reply on news.ycombinator.com