Hacker News Reader

Comment by Raphael | Hacker News Reader

My objection to it is that it would be trivial for email crawlers to decode the obfuscations most people use.

And yet: many email crawlers don't -- they're lazy and get plenty of email addresses just scraping nonobfuscated addresses.

So even trivial obfuscation turns out to help a lot. Here's someone's tests from a year ago:

http://techblog.tilllate.com/2008/07/20/ten-methods-to-obfus...

Every technique reduced spam by 60% or more, and even the dumb-as-rocks replacement of '@' and '.' with 'AT' and 'DOT' reduced the volume by size by over 99%.

Based on that analysis, javascript "scrambling" is the preferable method: It is 100%(1) effective and has no usability implications.

1: The analysis runs 1.5 years until July 2008 -- one must assume that crawlers has become more sophisticated. I.e. building the DOM, executing any javascript and then searching all visible text isn't that difficult, and less so now than in 2007.

It's still far from trivial if you're doing this on millions of pages, especially as you'll have to sandbox the JS in some way, which may or may not subtly break things in other ways. I suspect the effort isn't worth it.

Email harvesters probably don't want to run javascript because then they would be open to traps (like infinite loops or other cpu consuming scripts) that could be targeted at them.

Wouldn't the traps also make the web page unusable to regular users' browsers?

Such traps can be placed on decoy pages that users and good robots are unlikely to visit. Or, that legitimate users execute rarely -- when clicking a link to send a single legitimate email -- but email harvesters execute in excess.

There are ways to trick bots. 4chan used to have a second field named 'email' in their submission form that was hidden with the value set to "DO NOT PUT ANYTHING HERE" (or something similar) and spammers would blindly fill both email fields (unless someone was specifically targeting 4chan). I'll bet there are plenty of ways to get an email-crawler to click on some link that a normal person would not.

Especially obfuscations like "bob at gmail dot com". If I were a spambot I would just read until the first space and append "@gmail.com" and "@hotmail.com" and 5-6 other mail providers. This would break through 90% of people doing that, and they're high value targets too - they feel secure with their clever obfuscation and chances are their other anti-spam tools (and reflexes) are weaker.

But you're not a spam bot author. Based on what I've seen, most spam bot authors are pretty bad programmers.

Also, I would debate that these are high value targets. I'd wager that people who go out of their way to obscure their addresses are much less likely to purchase fake pills or fall for a phishing email than the average user.

With your method, you would have to get an email address after every word: in your own words "i would just read until the first space and append". At the first space ("bob "), you have no indication this is a valid prefix of an email address.

Secondly, I don't see how bob is a high value target. Someone who knows what spam is, and knows what a spam bot is, and knows they want to obfuscate it, is probably not someone who would make a purchase if they did receive spam.

I think that's one reason spam bot crawlers don't try that hard to obfuscate addresses: the recipients are of less value than those unobfuscated.

Yes. I've never bothered to obfuscate my email for exactly that reason, rather than for reasons of user convenience. If I were writing a program to crawl web pages for email addresses it would be extremely easy to include most of the common obfuscation patterns. If you really want to obfuscate then use some kind of image of the text, which is less easily machine readable.

Reply on news.ycombinator.com