Hacker News Reader

My experience with the RBL has been exclusively negative to date. We used to send out a mailing every couple of weeks, double opt-in and all that jazz. In spite of that there would always be some percentage of clueless people that forgot they'd signed up and that would go all vigilante on us, list us with every known spam blocker and so on. This would lead to password recovery emails not making it through any more (same server), tech support emails being lost and all kinds of other hassles, including direct loss of revenue because paying subscribers could not receive their payment confirmation emails.

In the end we gave up, email is a wonderful medium but if the reliability of the medium is left to people that are well willing to throw out the baby with the bathwater in a well meant but incompetent attempt at improving the situation then you're better off without it.

Blacklists are not the solution to any of these problems without a very rapid and transparent problem resolution process.

Spamhaus is one of the few such institutions that we've never had an issue with.

>My experience with the RBL has been exclusively negative to date.

>a well meant but incompetent attempt at improving the situation

I do not mean to dismiss your frustration, but I wonder if you are comparing the present situation to an unobtainable ideal.

Yes, email worked great in the early 1990s. (E.g., most people who ran the net-news infrastructure demanded that there be a working email address in the From field of every message, and I cannot recall anyone back then disagreeing with that demand.)

But the internet is drastically different now. We cannot just go back to the way it was.

Spam is here to stay. Mostly by virtue of the idiots that continue to buy products that are spamvertised.

Blocking the way the RBL does it is counterproductive, you can't combat spam effectively that way.

Spamhaus has the right idea (block the spam from zombies), the RBL and it's cousins are out of control.

If the battle with spam will ever really be won however it will be by a combination of end user education and legislation that will make it more expensive to send spam than the money you will make on it.

The education part of it will be the larger component, the legislation to mop up the remaining die hards.

To be fair - as the administrator on the receiving end of corporate email infrastructure - I pick and choose which blacklists and methods I'm going to use based on the needs of my users. If a list causes us to lose legitimate email consistently and we can't work with them, or they take up too much time, I won't use it. What we accept in our door is completely up to US, not any outsider.

As for sending email out- sure that's a problem -but we keep ourselves off blacklists (Which you CAN do - you simply avoid email marketing altogether, and stop annoying your users with anything but password resets and things they ask for in realtime - the rest you do through the web).

Yeah, I'm stunned too, especially since you've had so much more exposure than I have.

My first collision with RBLs was actually back in '97 or thereabouts. I was just starting to really mess around on the internet, had visions of becoming a dial-up ISP, all that good stuff. Anyway, my ISP at the time ended up on the MAPS RBL due to the actions of a customer-of-a-customer. Rather than give in, they decided to fight them, and I got stuck in the middle.

For various reasons, I didn't have to deal with RBLs again for quite a while (I worked for other/larger companies, where it was someone else's problem; I left the computer industry entirely for a few years; so on), but eventually I got to be a tech-support/sysadmin monkey for a local ISP. They occasionally found themselves blackholed by one list or another for various reasons, whether it was a root'd box or a customer doing something dumb, usually a mailing list. Still, dealing with RBLs happened at least monthly.

Since starting my own business, I've had to deal with RBLs probably 3 or 4 times a year. Not super often, but I don't have a really massive list of clients yet either.

As yet another data point, nearlyfreespeech.net refuses to send out email notifications for planned outages. One of the reasons they give is that some percentage of their customers will lazily or mistakenly flag their message as spam and they'll end up on an RBL, which they don't want to deal with.

I'm actually not opposed to RBLs being used as a factor in products like SpamAssassin. For one thing, SA can be configured by the end-user, which means that if they miss out on anything, they only have themselves to blame; and secondly, so long as the RBL isn't the sole reason for dropping a message or connection, it doesn't bother me nearly as much.

At the ISP I worked for, "missing email" was probably the number 1 complaint, or close to it, and we spent an awful lot of time trying to fine-tune things so that legitimate messages wouldn't be lost. So, color me equally surprised that you've had no trouble with it at all. :-)

"Since starting my own business, I've had to deal with RBLs probably 3 or 4 times a year. Not super often, but I don't have a really massive list of clients yet either."

So, this is more than I come in contact with RBL issues by a pretty good amount. I think about it occasionally because users ask how to tune their SpamAssassin rules regarding them, but that's usually not in response to any problem.

But, what I find interesting is that something you deal with 3 or 4 times a year is considered a big deal, and worth throwing out the whole RBL system over (which, as I noted, I do consider a reasonably useful addition to a spam filtering system). I can think of a significant number of things I have to deal with a lot more often on a mail server, even if I did have RBL troubles that often: bounces, software updates, spam filter tuning, fixing clamav (honestly, dealing with clamav accounts for probably 75% of my mail server related work), troubleshooting why a large mailing is going slow, user stuff, whatever. All of that stuff happens all the time. RBL issues are once in a blue moon, and usually triggered by some other issue that would have had to be dealt with anyway...getting listed in an RBL just made me aware of the problem sooner.

I've had to request de-listing twice that I can recall: Once for a new server that happened to end up on an IP that had been used by spammers. Once because of a pseudo-open relay (a web application had a hole that allowed mail to be sent to arbitrary recipients). In both cases it was fixed within the same day. I've also advised people on fixing open relays, DNS issues, among other things, over the years. I can't, at the moment, think of any false positives that I've dealt with directly or indirectly.

To add to the argument that RBLs aren't more trouble than they're worth, I'll mention a relatively recent issue in SpamAssassin that did effect a lot of our users, and our own mail server. There was a rule, distributed in the default RHEL/CentOS SpamAssassin package, that considered mail with a date in 20x0 to be too far in the future, and thus very spammy. When we ticked over into 2010, suddenly thousands of users weren't getting any mail (except a few spams that cleverly used dates in the distant past)...that's pretty dramatic. Easy to fix, but it effected a lot of people, and led to a couple dozen tickets and support queries in our forums. That one bug has caused more support request related to email than RBLs ever have. (And ClamAV tickets/queries still far outnumber any other aspect of our mail stack, even after the SpamAssassin 20x0 debacle.) We obviously aren't throwing out SpamAssassin just because a few problems pop up now and then.

Let me try to summarize this up in my final reply on the matter.

First, to the point that "3 or 4 times a year is not that much": At the moment I have in the neighborhood of a hundred active clients or so, many of them businesses. Having this particular issue come up with that small of an active client base is, to me, too much. Some of these businesses are research firms that are regularly sending data to outside contractors on tight deadlines and insurance companies trying to shuffle paperwork, contracts, and other agreements around.

Which brings me to my second point: we may simply have different ideas of what constitutes proper customer service. I believe, strongly, that it is totally unacceptable to have even one of my clients miss a deadline or experience even a relatively minor issue in their business because a mail server somewhere incorrectly flagged a message as spam, especially when that could easily have been avoided.

If the RBLs were managed more carefully, if there was accountability, if they had any incentive or desire whatsoever to go to great lengths to minimize the impacts on innocent users, then I would very likely have a different opinion of them. But, they do not. Vixie has been quite public in his very low opinion of spending any effort at all in minimizing the impact on innocent users. [1] We're talking about handing over massive amounts of network influence to a guy who would be willing to blackhole huge swaths of the internet just to decrease his personal spam volume by one or two messages per month. Is that really the internet you would like to be operating in? It's not the kind I want, for sure.

I don't see how the SpamAssassin issue you mentioned -- which I had to deal with also, albeit on a much much smaller scale -- makes RBLs any more or less trouble than they're worth. Either they're worth it, or they're not. I don't think they are, because I don't think they work well enough as a sole indicator of spammyness, and I think they have too many false positives. When they do have false positives, that translates to real, serious problems for real people.

Also, we're approaching this argument from two completely different environments. It's clear when you enumerate the list of problems that you have to spend time on that you're focusing mostly on things which bother a system administrator -- software updates and that sort of thing. However, I'm focusing on problems which bother users -- business owners, employees, individuals, professionals -- and while they don't tend to be impacted directly by things like software updates or slow mailing lists, they are directly impacted by things like all of their email to a contractor with whom they've had a longstanding relationship suddenly getting bounced by an erroneous RBL entry.

And, regardless of your experience with it, I'm telling you that it happens, and if my experience counts for anything at all, then it happens way too much.

[1]: For one quick example: http://www.mail-archive.com/nanog@merit.edu/msg23942.html

edit: For another example, http://www.mail-archive.com/nanog@merit.edu/msg04890.html and Vixie's reply ("zeal must become the norm.") at http://www.mail-archive.com/nanog@merit.edu/msg04904.html -- this is simply fundamentalism, and it will not lead to a healthy internet.

edit2: and more: http://www.networkworld.com/research/2001/0910feat.html ... ok, I'll quit now. Point is, there simply isn't grounds for saying, "it's not a problem."

edit3: http://www.infoworld.com/d/security-central/why-i-hate-rbls-...

"edit: For another example, http://www.mail-archive.com/nanog@merit.edu/msg04890.html and Vixie's reply ("zeal must become the norm.") at http://www.mail-archive.com/nanog@merit.edu/msg04904.html -- this is simply fundamentalism, and it will not lead to a healthy internet."

This is an 8 year old post. And, I think, it's turned out to be an accurate prediction.

ISPs no longer run open relays, ever. In 2002, an awful lot of them had open relays, and did nothing to prevent spam from their users. Likewise hosting providers; if the server owner was paying the bills, they'd keep the lights on for them, even if they were spewing millions of messages a day into the world.

SMTP authentication has become standard and well-supported by all mail clients and mail servers, and is easy to configure.

DNS is now a fundamental part of the spam prevention regime, and DKIM makes it moreso.

Spamming costs more today, relatively speaking, than it did eight years ago. Not just because of RBLs, but it certainly doesn't hurt. Spammers have to keep moving, constantly, in order to keep sending out their spam. Making it cost more is the only thing that works. Asking nicely doesn't do it.

"It's clear when you enumerate the list of problems that you have to spend time on that you're focusing mostly on things which bother a system administrator -- software updates and that sort of thing. However, I'm focusing on problems which bother users -- business owners, employees, individuals, professionals"

System administrators are the ones who have to fix any problems in the mail server. They see all the problems, not just one email going rogue.

Users also care about spam. Users care a lot about spam. Try turning off your spam filters altogether, and see how much happier your customers are. I bet they'll be really excited to know they aren't missing any mail (especially the 823 messages from a Nigerian prince, and the 348 messages about "テクニックに自信ありますか"--I have no idea what that translates to, it's just a spam in my spam folder right now).

RBLs do reduce spam, and make it harder for spammers to operate. I'll continue to use them (in a limited, weighted, manner) until DKIM or some other authentication mechanism makes them completely unnecessary. Spam is a serious problem, and it takes dramatic solutions to fix it.

It seems to me that all the hoopla has long since past. The problems that RBL maintainers were fighting against are no longer standard practice, and when you run into them it generally means the system administrator has done something wrong. False positives are certainly a bad thing, but they happen in all spam fighting systems.

Anyway, I wouldn't use them as a binary switch and I don't know many people that suggest doing so these days, but I certainly like having them available in SpamAssassin.

We obviously aren't throwing out SpamAssassin just because a few problems pop up now and then.

I did. In 2002 or 2003, they made a change which caused my then-users to lose mail. I can't remember the details now (something to do with rewriting mails too aggressively, leading to empty messages, and some changed defaults) but I still remember the shame of it, seven or eight years ago.

It was in a sense my fault, because I didn't test a new version nearly thoroughly enough.

I kept SpamAssassin available for my users, but I haven't used it myself since then, and don't plan to ever turn it on by default ever again.

Trust is hard-gained and easily lost.

7+ years ago. And I still feel ashamed.

First, as SwellJoe pointed out, a couple of times a year is not a big deal.

Blacklists have also proven quite effective in pressuring ISPs to clean up their act - they can still ignore bots/spammers/assorted morons, but they will soon find themselves unable to send e-mail, and hence, without legitimate customers. I admit there's some burden on the well-meaning, but it makes life a lot harder for spammers and spam-abiding ISPs.

So:

* 'my ISP at the time ended up on the MAPS RBL due to the actions of a customer-of-a-customer. (My ISP) decided to fight them'

* Then the place you worked at had: 'a root'd box (sic) or a customer doing something dumb, usually a mailing list'

That sounds similar to my own experience dpoing Unix work in the late 90s:

* Education provider has MS Exchange 5.0 box and is being used to send out gigs of porno spam (there is no relay control at all built into Exchange 5.0, MS made everyone pay to buy Exchange 5.5)

* Various boxes running Sendmail and incorrectly configured (fairly easy on sendmail).

In both my and your cases, the impression I got of RBL was positive. They Blacklist found real abuse, and the RBL's users/customers got what they wanted: less spam.

I'm kinda stunned you have such a strong emotional feeling about them.

The lists themselves don't bother me so much. The people behind them, though, scare the bejesus out of me.

There's a line in Nancy Kress' novel Beggars in Spain, where one of the main characters says, of another (a woman whose paranoia has driven her to do some fairly extreme things), that she would nuke the world if it would finally make her feel safe. Too often, that quote floats into my mind when reading things like the SPEWS FAQ.

Well if you want an example of what goes wrong, just read the reddit announcement where they talk about a lot of their customers who couldn't receive their email because Trend Micro had decided that the entire amazon cloud was a dial-up pool.

Frankly black lists, grey lists and spammers should just go to hell.

"black lists, grey lists and spammers should just go to hell"

Spammers first though. Otherwise things will get worse before they get better.

Not necessary, we can just punish those whom they sell services for - then there will be no incentive to spam.