Hacker News Reader

Comment by a5seo | Hacker News Reader

I found this to work very well.

Only thing is you need to append keys randomly, say one out of 5 requests, in different locations if possible. And you should hand out 20 or so different unique strings.

Then use google alerts to find them republishing your stuff an send DMCA takedown notices with Google (don't send them a c&d first so you have the element of surprise and maximum chance of causing real pain). If they don't take care of the DMCA issue with Google, Google will remove their pages from the index. They won't know why but their site will slowly die.

View on HN

google alerts = clever.

the other thing to do is include some some encoding of the served-to IP address (maybe md5 of the first and last half of the IP?)

To generate the unique string, do SHA1(content + TheirIP + unique secret). The chance of collision is practically zero. Also, you can then conclusively prove that it was you who created this unique string - which may come in handy.

(Talking about SHA1 because I used it myself in a lighter version - just a hash of my name - to check how the search engines work with the content of my little blog).

Not true. See http://en.wikipedia.org/wiki/Birthday_problem

The birthday problem basically just cuts the number of bits in half. For SHA1 this changes from a chosen collision at absurd odds to a random collision at merely ridiculous odds.

I like it, but you need to do this in a way that you append a known/deterministic string so you can monitor THAT one in Google Alerts.

I don't care whether the unique strings collide (although I love the idea of being able to tie the IP address of the crawler to the page on which the content was published for maximum evidence collection), I just want Google Alerts to find them reliably AND I don't want the spammer to catch on and be able to easily remove the unique strings before I catch him.

The only way for him to find my little landmines (to strip them) would be to read through every piece of content he scraped from me.

By the way - thinking of all this - why would not you turn this scraper into a free advertisement for you ? If you can detect when it crawls your site, just insert randomly the backlinks to your site into the content you give out to them.

This way you will want that they steal more from you :-)

(how to put these backlinks in a way that would be difficult to remove - is another story. But if you use other links, e.g. tinyurl, in your material - then you could probably use that :-) Still fairly simple to remove but would require more work from them.)

Reply on news.ycombinator.com