Hacker News Reader

Comment by _urga | Hacker News Reader

I remember reading pg's post, it was a classic.

Anyone here working on anti-spam?

Where are things at today?

From my perspective (~500 employee mail server), greylisting had a much larger impact at the time, thanks to the spambots/viruses attempting direct connection to mail servers. Extremely effective, zero false positives, much lighter on resources. I did use both, of course, so that I could keep a record of how effective the systems were.

Today the situation has flipped. Most of the spam we get is coming from authoritative servers (ie: gmail, yahoo, etc), making stuff like SPF/DKIM/etc next to worthless from a spam perspective (it's still marginally useful for forgeries), while bayes (or in general, trainable) filters are essentially the only thing that can differentiate it reliably.

With a modern setup, you can basically next to zero spam and no false positives. In fact, honest email marketing (ie: mailing lists you've actually subscribed to) are from my experience the only thing that throws these filters off.

Thanks, one thing we also found is that spammers tend to be poor at RFC standards, in a way that Gmail etc. will have no problem with, but which are obviously broken.

For example, we use our own https://github.com/ronomon/mime to detect and reject email which has missing multi-parts (no terminating boundary delimiter). All of this has been spam so far, and we are yet to see a false positive. I don't think SpamAssassin has a rule for this (yet)?

Another example is illegal header characters, which are almost always spam, with a handful of false positives (usually machine-generated).

That is an interesting approach. Care to let us know how you go from https://github.com/ronomon/mime to some kind of SMTP server plugin (like for postfix for example)?

Thanks, you might find Haraka to be easiest since it's already Javascript.

Postfix may require a process callout, you might need to write a milter.

I agree that greylisting was the cat's meow back then. I setup a VM running CentOS with Postfix and Postgrey as a MTA for our "work" email server and the result was a massive reduction in spam.

I was running my own email server round about this time, and gave it up mid-2000s, in great part because of spam. Both inbound filtering and outbound deliverability into other people's filters.

It seems that centralisation has been highly effective in spam-fighting. Google must have a huge corpus of spam and ham, and they also have the benefit of being able to spot patterns of incoming mail "live" across a very large number of accounts.

I'm not directly working on spam, but it seems like there's a heavy reliance on ipv4 address reputation these days, which is probably one reason mailservers won't use ipv6 any time soon.

Address reputation was always big, even back then.

In the last years though, I cannot really recommend to use any of the DNSBL anymore. I've encountered more cases where legitimate servers were blocked due to netblock vicinity or indeed previous ownership than actual spam issues.

Greylisting will still catch dynamic allocations almost as effectively, while you won't reject legitimate mail due to server and/or DNSBL issues.

Have you tried using a quorum of DNSBL (e.g. barracudacentral.org, cbl.abuseat.org, truncate.gbudb.net) to reduce false positives?

In other words, if at least two DNSBL queries agree, then reject, or feed this information to the rest of the spam pipeline?

This is pretty common, and systems like SA do this for you by batching responses and calculating a score.

I found this to be pretty much worthless if you already have greylisting, even for high-quality curated lists such as spamhaus SBL/XBL.

Not "working" on anti-spam, but for my family's email server, we run rspamd and it works really well.

Reply on news.ycombinator.com