1. File a report with Amazon. 2. Block the IPs using iptables 3. File a DMCA complaint against them with Amazon 4. Send a cease and desist letter to them or offer to sell them data.
1. File a report with Amazon. 2. Block the IPs using iptables 3. File a DMCA complaint against them with Amazon 4. Send a cease and desist letter to them or offer to sell them data.
We blocked all the EC2 IP ranges (much as I hated to do so; and did you know how big a chunk of the Internet that is?). They responded by re-routing through anonymizers, especially TOR. We could play whack-a-mole with that, but it's a manpower-intensive arms race.
File a DMCA complaint against them with Amazon
Can you explain to me how DMCA comes into this? I think that, although there's nothing specific in our T&C (yet) disallowing this, the fact that it's arguably at a DoS level might push this into criminal territory.
Is this a continuation of this: http://news.ycombinator.com/item?id=2085859
If so, did you try the iptables rate limiting as described? Or the Varnish rate limiting?
Rate limiting is difficult, because our legitimate users are frequently Fortune 50 companies coming through a proxy. It's hard to differentiate that heuristically. Based on the fingerprints we've observed, we might be able to hard-code something, but if the bad guys change the factors we've picked out, it would become whack-a-mole again. Also, at this level, the fact that we've got multiple servers in a farm works against recognition.
The kind of crawling they're doing isn't going to be helped by caching. They're looking for product pricing, and we have customized pricing for every customer (Fortune 500 customers want their own specific contracts), so we wouldn't be getting any cache hits.
I take it you don't do any sort of authentication which you could use to differentiate your real customers from this actor?
Do you have any large binary files you could serve up once you've identified the actor?